Positioning method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides a positioning method, device and equipment, a computer readable storage medium and a computer program product. The method comprises steps that a first ground texture image is acquired, a pre-constructed texture map is acquired, and the texture map comprises a plurality of reference ground texture images and reference positioning information of each reference ground texture image; determining a second ground texture image satisfying a position relation condition with the first ground texture image from the plurality of reference ground texture images; determining a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image; and determining first positioning information of the electronic equipment based on the conversion matrix and the reference positioning information of the second ground texture image. According to the invention, the accuracy and continuity of positioning can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a positioning method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] Positioning refers to determining the position and posture of an electronic device in the environment, usually in two-dimensional or three-dimensional space. The positioning process involves the use of sensors, algorithms, and environmental information to estimate and update the coordinates of the electronic device and its orientation in space. Positioning is a key step in the fields of robot autonomous navigation, unmanned driving, industrial automation, etc.

[0003] In the related art, positioning is performed based on a map or a deployed QR code. When positioning is performed based on a map, the related art has the problem of inaccurate positioning; and when positioning is performed based on a QR code, the related art has the problem of high positioning cost and inability to perform continuous and accurate positioning. Summary of the invention

[0004] The embodiments of the present application provide a positioning method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy and continuity of positioning.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a positioning method, the method comprising:

[0007] Collecting a first ground texture image and obtaining a pre-built texture map, wherein the texture map includes a plurality of reference ground texture images and reference positioning information of each of the reference ground texture images;

[0008] Determine, from the plurality of reference ground texture images, a second ground texture image that satisfies a position relationship condition with the first ground texture image;

[0009] determining a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image;

[0010] Based on the conversion matrix and the reference positioning information of the second ground texture image, first positioning information of the electronic device is determined.

[0011] The present application provides a positioning device, the device comprising:

[0012] A first acquisition module is used to acquire a first ground texture image and obtain a pre-built texture map, wherein the texture map includes a plurality of reference ground texture images and reference positioning information of each of the reference ground texture images;

[0013] A first determining module, configured to determine, from the plurality of reference ground texture images, a second ground texture image that satisfies a position relationship condition with the first ground texture image;

[0014] A second determining module, configured to determine a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image;

[0015] The third determination module is used to determine the first positioning information of the electronic device based on the conversion matrix and the reference positioning information of the second ground texture image.

[0016] An embodiment of the present application provides a computer device, including:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the positioning method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the positioning method provided in the embodiment of the present application when being executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the positioning method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] In the embodiment of the present application, the conversion matrix between the first ground texture image and the second ground image is determined based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, that is, the embodiment of the present application performs image registration based on the frequency domain space. Since the noise is concentrated in the high frequency part of the frequency domain, and the illumination change is reflected in the low frequency part of the frequency domain, the noise and illumination change can be filtered out during the image registration process, thereby improving the accuracy of image registration, and then improving the accuracy of positioning. In addition, the image to be registered in the embodiment of the present application is a ground texture image, rather than an image on the ground that is easily changed. Since the ground texture image has the characteristics of small frequency and small amplitude of change, it can improve the stability of positioning in dynamic scenes. In addition, since the electronic device can collect the first ground texture image in real time during the positioning process, and then perform positioning based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, the purpose of continuous positioning can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a network architecture of a positioning system 100 provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a first process of a positioning method provided in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a process for determining a rotation matrix provided in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of a process for determining a scale factor and a rotation factor provided in an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a process for determining a displacement vector provided in an embodiment of the present application;

[0029] Figure 7 is a second flow chart of the positioning method provided in an embodiment of the present application;

[0030] Figure 8 This is a third flow chart of the positioning method provided in an embodiment of the present application.

[0031] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) Simultaneous Localization and Mapping (SLAM) refers to the robot building a map of the environment and locating its own position in an unknown environment at the same time. The SLAM system uses laser radar to scan the environment in real time, gradually build a map, and determine its own position in the newly created map.

[0040] 2) Visual positioning refers to using cameras or other visual sensors to obtain environmental information and determine its own position in the environment through image processing and computer vision technology. Visual positioning is widely used in fields such as robots, self-driving cars, and augmented reality (AR) devices. Visual positioning usually involves steps such as image acquisition, image processing, feature extraction, feature matching, positioning calculation, and map construction.

[0041] 3) Extended Kalman Filter (EKF) is an algorithm for state estimation of nonlinear systems. It recursively predicts and updates the state by linearizing the nonlinear state transfer equation and observation equation of the system through first-order Taylor expansion. The specific process includes prediction and update, where: the prediction process uses the state transfer equation of the system to predict the state and covariance matrix at the next moment. The update process combines the observation value and calculates the Kalman gain to update the state estimate and covariance.

[0042] 4) Fourier-Mellin Transform (FMT) is a mathematical tool that combines Fourier transform with Mellin transform and is often used in image processing. By converting the image from the spatial domain to the frequency domain and performing a polar coordinate transformation on the image, FMT can be invariant to geometric transformations such as rotation, scaling, and translation of the image.

[0043] 5) Image registration is the process of aligning an image (usually a new or unprocessed image) with another image in a known coordinate system. This process is very important in computer vision, medical image processing, remote sensing, robotic navigation and other fields. The purpose of image registration is to find a transformation (such as rotation, scaling, translation and / or distortion) so that corresponding feature points in the two images can be accurately aligned. The steps of image registration usually include feature detection, feature matching, transformation estimation, transformation application and result verification.

[0044] 6) Time domain images refer to image sequences that change in the time dimension, which are used to describe the changes of images over time. Each frame of the image represents a scene or state at a certain moment. Time domain images have the characteristics of time correlation and dynamic changes.

[0045] 7) Frequency domain image refers to the representation of an image after converting it from the spatial domain to the frequency domain through Fourier transform. The frequency domain image describes the distribution of frequency components in the image rather than the brightness value of the pixels.

[0046] 8) Phase correlation matrix is ​​a matrix based on the frequency domain phase information of the image, which is used to detect the translation, rotation and scale transformation between images. By determining the phase difference and detecting the peak position in the matrix, the alignment between images can be accurately determined.

[0047] 9) Ground texture map refers to a map that helps the robot to locate and navigate by sensing and recording the texture features of the ground surface, where the texture features can be colors, patterns, etc. The ground texture map records the ground texture information on the robot's driving path and can be used as a positioning reference, allowing the robot to locate by comparing the ground texture obtained by the current sensor with the pre-generated texture map.

[0048] In order to better understand the positioning method provided in the embodiments of the present application, the positioning method in the related art and its existing shortcomings are first described.

[0049] The AGV robot is a common automated handling robot in factory scenes, which is used in logistics and transportation scenes. In related technologies, common positioning methods for AGV robots include laser SLAM positioning and visual positioning. Visual positioning includes traditional visual localization and mapping (VLSAM) methods and QR code positioning methods. Laser SLAM and visual SLAM positioning methods require pre-map construction, and the positioning effect depends on the mapping effect. If the scene changes significantly, the map needs to be updated, otherwise wrong matching information will be generated during the positioning process. The QR code positioning method is not affected by environmental changes, but it is necessary to transform the on-site environment before the robot enters the scene to run, and deploy a large number of QR codes in the scene so that the robot can directly obtain global positioning information when the QR code is detected during operation.

[0050] The characteristic of logistics and transportation scenarios is that there will be frequent and drastic changes in the scene, so that the positioning algorithm that relies on the factory environment mapping results needs to constantly update the map to ensure the accuracy of positioning, which is impossible on low-computing AGVs. The QR code positioning method greatly increases the deployment cost because it requires a significant transformation of the environment, and the QR code method cannot provide observation information between two QR codes.

[0051] Combining the above-mentioned related technologies, it can be seen that the disadvantages of the related technologies include at least: first, low positioning accuracy; second, a complex positioning process and inability to achieve continuous and accurate positioning.

[0052] The embodiments of the present application provide a positioning method, apparatus, computer equipment, computer-readable storage medium, and computer program product to improve the accuracy and continuity of positioning. The following describes an exemplary application of the computer equipment provided in the embodiments of the present application. The computer equipment provided in the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, unmanned vehicles, drones, robots, and aircraft, and can also be implemented as servers. Below, an exemplary application when the computer equipment is implemented as a terminal will be described.

[0053] See also Figure 1 , Figure 1 It is a schematic diagram of the network architecture of the positioning system 100 provided in an embodiment of the present application. To support a positioning application, the terminal 400 is connected to the server 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0054] The terminal 400 may be an unmanned vehicle, an unmanned aerial vehicle, a robot, an aircraft, etc. The terminal 400 is used to periodically send a map acquisition request to the server 200. The server 200 is used to send a pre-built texture map to the terminal 400 based on the map acquisition request, wherein the texture map includes multiple reference ground texture images and reference positioning information of each reference ground texture image; the terminal 400 is also used to collect a first ground texture image, and determine a second ground texture image that satisfies a positional relationship condition with the first ground texture image from the multiple reference ground texture images; determine a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image; determine the first positioning information of the electronic device based on the conversion matrix and the reference positioning information of the second ground texture image.

[0055] In some embodiments, the terminal 400 may periodically send a map acquisition request to the server 200, and the server 200 may determine whether there is an updated texture map based on the map acquisition request, and if there is an updated texture map, send the updated texture map to the terminal 400. The server 200 may also actively send the updated texture map to the terminal 400 when the texture map is updated.

[0056] In some embodiments, the pre-built texture map may be built in advance by the terminal 400, and the pre-built texture map is stored in the terminal 400. Based on this, the terminal 400 can obtain the pre-built texture map from its own storage system without obtaining the pre-built texture map from the server 200, thereby improving the positioning speed.

[0057] In the embodiment of the present application, the terminal 400 determines the conversion matrix between the first ground texture image and the second ground image based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, that is, the embodiment of the present application performs image registration based on the frequency domain space. Since the noise is concentrated in the high frequency part of the frequency domain, and the illumination change is reflected in the low frequency part of the frequency domain, the noise and illumination change can be filtered out during the image registration process, thereby improving the accuracy of image registration, and then improving the accuracy of positioning. In addition, the image to be registered in the embodiment of the present application is a ground texture image, rather than an image on the ground that is easily changed. Since the ground texture image has the characteristics of small frequency and small amplitude of change, it can improve the stability of positioning in dynamic scenes. In addition, since the electronic device can collect the first ground texture image in real time during the positioning process, and then perform positioning based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, the purpose of continuous positioning can be achieved.

[0058] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0059] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0060] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0061] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0062] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0063] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0064] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0065] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0066] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0067] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0068] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0069] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The positioning device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first acquisition module 4551, a first determination module 4552, a second determination module 4553 and a third determination module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0070] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the positioning method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0071] In some embodiments, the terminal can implement the positioning method provided in the embodiments of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a positioning APP, an automatic transportation APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0072] The positioning method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0073] The following describes the positioning method provided by the embodiment of the present application. As mentioned above, the computer device implementing the positioning method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0074] It should be noted that in the examples of the positioning method below, the electronic device is a robot. Based on the understanding of the following, those skilled in the art can apply the positioning method provided in the embodiments of the present application to the positioning of other types of electronic devices such as drones and unmanned vehicles. The embodiments of the present application can also be applied to various scenarios, including but not limited to logistics and transportation, agriculture, instant messaging, artificial intelligence, smart transportation, assisted driving, etc.

[0075] See also Figure 3 , Figure 3 is a first flow chart of the positioning method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are used to illustrate the positioning method provided in the embodiment of the present application. Figure 3 The execution subject is the terminal.

[0076] In step S101, a first ground texture image is collected, and a pre-built texture map is obtained.

[0077] In the embodiment of the present application, an image acquisition module is deployed on the electronic device, and during the positioning process of the electronic device, the first ground texture image can be acquired in real time through the image acquisition module. The image acquisition module can include at least one of a visible light camera, an infrared camera, a structured light camera, and a dual-camera stereo vision.

[0078] In other embodiments, the first ground texture image may contain rich information such as ground color, brightness, roughness, and geometric shape. The color and brightness changes of the ground can reflect different materials and surface conditions, such as cement, sand, and tiled floors. The roughness and texture information of the ground can distinguish different ground types, such as smooth tiled floors and rough gravel floors. The geometric shape features of the ground can be cracks, protrusions, depressions, etc.

[0079] In an embodiment of the present application, the texture map includes multiple reference ground texture images and reference positioning information of each reference ground texture image. The texture map may be constructed in advance by an electronic device, and the reference positioning information is used to characterize the spatial position of the reference ground texture image in the environmental space. Since different reference ground texture images are located at different spatial positions, different reference ground texture images correspond to different reference positioning information. In some embodiments, the reference positioning information may include position coordinates, direction information, or relative position and direction information. Since the reference ground texture image itself does not have directionality, the direction information of the reference ground texture image can be represented by a preset initial value. At this time, the reference positioning information can be considered as a 3 by 1 matrix. The first row of matrix elements and the second row of matrix elements of the matrix represent the position information of the reference ground texture image, and the third row of matrix elements of the matrix represent the direction information of the reference ground texture image. Exemplarily, the reference positioning information can be or

[0080] In step S102, a second ground texture image that satisfies a positional relationship condition with the first ground texture image is determined from a plurality of reference ground texture images.

[0081] In an embodiment of the present application, when the distance between the location of the reference ground texture image and the location of the first ground texture image is less than a preset distance threshold, it is determined that the reference ground texture image and the first ground texture image satisfy the positional relationship condition; when the distance between the location of the reference ground texture image and the location of the first ground texture image represents that the reference ground texture image belongs to N reference images near the first ground texture image, it is determined that the reference ground texture image and the first ground texture image satisfy the positional relationship condition.

[0082] In some embodiments, the above step S102 can be implemented by the following steps S1021 to S1023, which are described in detail below.

[0083] In step S1021, second positioning information is determined based on the first time domain image of the first ground texture image.

[0084] Here, the first time domain image can be an image sequence of the first ground texture image in the time domain space, and the image similarity between the first time domain image and each reference ground texture image can be determined based on the first time domain feature of the first time domain image and the second time domain feature of each reference ground texture image, wherein the first time domain feature is a feature obtained by extracting features from the first time domain image; the second time domain feature is a feature obtained by extracting features from the image sequence of the reference ground image in the time domain space; and then the reference ground texture image corresponding to the image similarity with the largest value is determined as the similar ground texture image, and the reference positioning information of the similar ground texture image is determined as the second positioning information. The second positioning information can be considered as an estimate of the positioning information for the first time domain image.

[0085] In some embodiments, a positioning module may also be deployed in the electronic device, based on which the positioning module receives signals from satellites and uses these signals to determine the second positioning information of the device. Exemplarily, the Global Positioning System (GPS) module receives signals from multiple satellites and combines the time difference and phase information of these signals to determine the location of the electronic device. This technology relies on GPS, which uses a group of satellite networks to provide accurate location information worldwide.

[0086] In step S1022, based on the second positioning information and the reference positioning information of each reference ground texture image, a first distance between a first position corresponding to the first ground texture image and a reference position corresponding to each reference ground texture image is determined.

[0087] In the embodiment of the present application, the first position may refer to the position in the environment space where the first ground texture image is located. In some embodiments, the first position may represent a point, in which case the first position may be the position in the environment space where the center point of the first ground texture image is located, and the first position may be represented as a coordinate; the first coordinate may also represent an area, in which case the first position is the area in the environment space where the first ground texture image is located, and when the area is a rectangular area, the first position may be represented as the coordinates of the four vertices of the area.

[0088] In addition, the reference position may refer to the position in the environment space where the reference ground texture image is located. In some embodiments, the reference position may represent a point, in which case the reference position may be the position in the environment space where the center point of the reference ground texture image is located, and the reference position may be represented as a coordinate; the reference coordinate may also represent an area, in which case the reference position is the area in the environment space where the reference ground texture image is located, and when the area is a rectangular area, the reference position may be represented as the coordinates of the four vertices of the area.

[0089] Based on this, when the first position and the reference position both represent a point, the distance between the two points is determined as the first distance; when the first position represents a point and the reference position represents an area, or when the first position represents an area and the reference position represents a point, the shortest distance from the point to the area can be determined as the first distance; when the first position and the reference position both represent an area, the shortest distance between the two areas is determined as the first distance.

[0090] In step S1023, the reference ground texture image corresponding to the first distance less than the distance threshold is determined as the second ground texture image that satisfies the positional relationship condition with the first ground texture image; alternatively, the first distances are sorted in ascending order to obtain the sorted first distances; and the reference ground texture images corresponding to the first N first distances in the sorted first distances are determined as the second ground texture images that satisfy the positional relationship condition with the first ground texture image.

[0091] In the embodiment of the present application, N is a positive integer. N can be a value set in advance. For example, the value of N can be 2, 3, 5, etc.

[0092] In some embodiments, the distance threshold may be a value set in advance, and the distance threshold may be used to screen out a second ground texture image that satisfies a positional relationship condition with the first ground texture image from a plurality of reference ground texture images. Exemplarily, the value of the distance threshold may be 0.1 meter, 0.2 meter, 0.5 meter, 1 meter, etc.

[0093] The embodiment of the present application provides two implementation methods for determining the second ground texture image, wherein the first implementation method is: from each first distance, determine a second distance that is less than a distance threshold, and since there is a one-to-one correspondence between the reference ground texture image and the first distance, then filter out the ground texture image corresponding to the second distance from multiple reference ground texture images, and determine the ground texture image corresponding to the second distance as the second ground texture image that satisfies the position relationship condition with the first ground texture image. In this way, at least one second ground texture image is determined.

[0094] The second implementation method is: sorting the first distances in ascending order from small to large to obtain sorted first distances; then selecting reference ground texture images corresponding to the first N first distances among the sorted first distances from multiple reference ground texture images; and determining the reference ground texture images corresponding to the first N first distances among the sorted first distances as second ground texture images that meet the position relationship condition with the first ground texture image. In this way, N second ground texture images are determined.

[0095] Through the above steps S1021 to S1023, a second ground texture image that satisfies the positional relationship condition with the first ground texture image can be quickly and accurately screened out from multiple reference ground texture images. The second ground texture image can be a reference image close to the first ground texture image.

[0096] Continue to see Figure 3 , and the description continues with step S102 above.

[0097] In step S103, a conversion matrix between the first ground texture image and the second ground texture image is determined based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image.

[0098] In the embodiment of the present application, image registration is performed based on two frequency domain images, so as to determine a conversion matrix between the first ground texture image and the second ground texture image.

[0099] In some embodiments, the first ground texture image collected by the electronic device can be an image in the time domain space. Before executing the above step S103, the first frequency domain image of the first ground texture image can also be determined by discrete Fourier transform, fast Fourier transform, etc.; similarly, the second frequency domain image of the second ground texture image can also be determined by discrete Fourier transform, fast Fourier transform, etc.

[0100] In some embodiments, frequency domain image registration refers to the process of aligning two or more images using the frequency domain information of the images. Different from the traditional spatial domain-based methods, frequency domain image registration converts the images into the frequency domain and uses the features in the spectrum for matching and alignment, which is usually achieved by Fourier transform.

[0101] The steps of frequency domain image registration include: (1) Fourier transform. First, Fourier transform the image to be registered to convert the image from the spatial domain to the frequency domain. The frequency domain representation can decompose the image into sine and cosine components of different frequencies. (2) Spectral feature matching. In the frequency domain, image transformations such as displacement, rotation, and scaling will affect the phase information of the spectrum. For example, image translation is manifested as a linear change in phase in the frequency domain, while rotation and scaling are manifested as angle and scale changes in the spectrum. (3) Frequency domain registration is performed using the phase correlation method. The phase correlation method estimates the amount of image translation by calculating the phase difference in the frequency domain representation of the two images. After Fourier transform, the phase difference between the two images is directly related to their geometric transformation. Therefore, the scale change, rotation, and translation between the images can be accurately determined through phase information to obtain the scale factor, rotation factor, and translation vector. (4) Inverse transformation and resampling: After determining the geometric transformation of the image (such as translation, rotation or scaling), one of the images is geometrically corrected according to the determined transformation, and the image is converted from the frequency domain back to the spatial domain using an inverse Fourier transform. Next, the image is resampled to align it with the other image.

[0102] In the embodiment of the present application, a scale matrix, a rotation matrix and a translation matrix can be determined based on the scale factor, the rotation factor and the translation vector respectively; and then the product of the scale matrix, the rotation matrix and the translation matrix is ​​determined as a transformation matrix.

[0103] In some embodiments, see Figure 4 The above step S103 can be implemented by the following steps S1031 to S1035, which are described in detail below.

[0104] In step S1031 , a scale factor and a rotation factor are determined based on the first frequency domain image and the second frequency domain image.

[0105] In some embodiments, the phase correlation of the two frequency domain images may be used to determine the scale factor and the rotation factor, wherein the scale factor is used to characterize the degree of scaling of the image, and the rotation factor is used to characterize the rotation direction and rotation angle of the image. That is, the scale factor is used to enlarge or reduce the image, and the rotation factor is used to rotate the image.

[0106] In some embodiments, see Figure 5 The above step S1031 can be implemented by the following steps S0311 to S0314, which are described in detail below.

[0107] In step S0311, coordinate transformation is performed on the first frequency domain image to obtain a fourth frequency domain image of the first ground texture image in a logarithmic polar coordinate system.

[0108] Here, the first frequency domain image is a frequency domain image in a Cartesian coordinate system. In a log-polar coordinate system, scale changes can be converted into horizontal translations, and rotation changes can be converted into vertical translations. Based on the coordinate system conversion formula, the first frequency domain image is transformed to obtain a fourth frequency domain image of the first ground texture image in the log-polar coordinate system.

[0109] In some embodiments, the radial direction in polar coordinates is expressed as a logarithmic radial direction, see formula (1); and the angle in polar coordinates is expressed as an angle as shown in formula (2).

[0110]

[0111] In formulas (1) and (2), u, v are the coordinates of the frequency domain image, and (ρ, θ) are the coordinates in the logarithmic polar coordinate system.

[0112] In step S0312, coordinate transformation is performed on the second frequency domain image to obtain a fifth frequency domain image of the third ground texture image in a logarithmic polar coordinate system.

[0113] In some embodiments, the implementation process of step S0312 is similar to the implementation process of the above-mentioned step S0311, and therefore, the implementation process of step S0312 may refer to the implementation process of the above-mentioned step S0311.

[0114] In step S0313, a first phase correlation matrix between the first ground texture image and the third ground texture image is determined based on the fourth frequency domain image and the fifth frequency domain image.

[0115] In some embodiments, first, the fourth frequency domain image may be Fourier transformed to obtain a sixth frequency domain image; the fifth frequency domain image may be Fourier transformed to obtain a seventh frequency domain image; then the conjugate phase product of the sixth frequency domain image and the seventh frequency domain image may be determined; finally, the conjugate phase product may be inverse Fourier transformed to obtain a first phase correlation matrix between the first ground texture image and the third ground texture image.

[0116] In the implementation process of the above step S0313, a second Fourier transform is performed on the image after log polar coordinate conversion, so that the rotation and translation of the image will be reflected as a simple phase difference in the frequency domain. The purpose of the second Fourier transform here is to convert the rotation and scale transformation into a linear translation in the frequency domain, so that the scale transformation and rotation transformation between the first ground texture image and the third ground texture image can be determined by a simple phase correlation.

[0117] In step S0314, a scale factor and a rotation factor are determined based on the peak position of the first phase correlation matrix.

[0118] In some embodiments, the scale factor may be determined by the following formula (3), and the peak position and rotation factor of the first phase correlation matrix may be determined by the following formula (4).

[0119] s=e Δρ Formula (3);

[0120] (Δρ,Δθ)=argmax(r(x,y)) Formula (4);

[0121] In formulas (3) and (4), s represents the scale factor, Δθ represents the rotation factor; argmax is a function representing the value of the independent variable corresponding to the maximum value of r(x, y); (Δρ, Δθ) refers to the peak position of the first phase correlation matrix, where the peak position refers to the position of the maximum matrix element, which can be obtained by the function argmax.

[0122] Through the above steps S0311 to S0314, based on two Fourier transforms, the rotation and translation of the image can be reflected as a simple phase difference in the frequency domain. Based on this, the scale factor and the rotation factor can be quickly and conveniently determined through the peak position of the first phase correlation matrix.

[0123] Continue to see Figure 4 , and the description continues with step S1031 above.

[0124] In step S1032, the first ground texture image is scaled using a scale factor to obtain a third ground texture image.

[0125] In the embodiment of the present application, a scale factor greater than 1 indicates an enlarged image, and a scale factor less than 1 indicates a reduced image. For example, assuming that the scale factor is 2, the first ground texture image is enlarged by 2 times.

[0126] In some embodiments, the first ground texture image is first enlarged or reduced based on the scale factor to obtain a third ground texture image; then, according to the pixel position of the third ground texture image, the corresponding pixel value is determined from the first ground texture image by interpolation, wherein the interpolation method can be any one of nearest neighbor difference, bilinear interpolation, and bicubic interpolation. Taking the interpolation method as nearest neighbor difference as an example, the pixel value of the nearest integer coordinate in the first ground texture image is directly determined as the pixel value of the non-integer coordinate in the third ground texture image; finally, the value of each pixel in the third ground texture image is obtained by interpolation, thereby generating a third ground texture image of a new size.

[0127] In step S1033, the third ground texture image is rotationally transformed using the rotation factor to obtain a fourth ground texture image.

[0128] In the embodiment of the present application, when the rotation factor is positive, it indicates counterclockwise rotation; when the rotation factor is negative, it indicates clockwise rotation. For example, a rotation factor of 90 degrees indicates that the third ground texture image is rotated 90 degrees counterclockwise.

[0129] In some embodiments, the image center point of the third ground texture image can be determined as the rotation center; then a transformation matrix is ​​generated based on the rotation factor, and the transformation matrix is ​​a two-dimensional matrix; then, the third ground texture image is rotated based on the transformation matrix to obtain a rotated texture image; then, the transformed pixel value is determined by nearest neighbor interpolation, bilinear interpolation or bicubic interpolation; finally, the fourth ground texture image can be obtained by keeping the image size unchanged and cropping the image portion that exceeds the boundary.

[0130] In step S1034, a displacement vector is determined based on the third frequency domain image and the second frequency domain image of the fourth ground texture image.

[0131] In the embodiment of the present application, the displacement vector between the corresponding time domain images can be determined by the phase correlation matrix of the frequency domain image. The displacement vector can describe the vector of the translation difference between the two images, and the displacement vector indicates the distance that an image moves in the horizontal and vertical directions respectively.

[0132] In some embodiments, see Figure 6 The above step S1034 can be implemented through the following steps S0341 and S0342, which are described in detail below.

[0133] In step S0341, a second phase correlation matrix between the fourth ground texture image and the third ground texture image is determined based on the third frequency domain image and the second frequency domain image.

[0134] In some embodiments, the method for determining the second phase correlation matrix in the above step S0341 is similar to the method for determining the direction of the first phase correlation matrix in the above step S0313. Based on this, the implementation process of the above step S0341 can refer to the implementation process of the above step S0313.

[0135] In step S0342, a displacement vector is determined based on the peak position of the second phase correlation matrix.

[0136] In the embodiment of the present application, the peak position of the second phase correlation matrix is ​​directly determined as the displacement vector.

[0137] Through step S0341 and step S0342, the displacement vector can be accurately and quickly determined based on the peak position of the second phase correlation matrix.

[0138] Continue to see Figure 4, and the description continues with step S1034 above.

[0139] In step S1035, a transformation matrix is ​​determined based on the scale factor, the rotation factor and the displacement vector.

[0140] In an embodiment of the present application, a scale matrix, a rotation matrix and a displacement matrix may be constructed respectively based on the scale factor, the rotation factor and the displacement vector; and then the product of the scale matrix, the rotation matrix and the displacement matrix may be determined as a transformation matrix.

[0141] In some embodiments, the above step S1035 can be implemented by the following steps S0351 to S0354, which are described in detail below.

[0142] In step S0351, a scaling matrix is ​​constructed using the scaling factor, the preset first value and the preset second value.

[0143] In the embodiment of the present application, the preset first value and the preset second value may be values ​​set in advance. For example, the preset first value may be 1, and the preset second value may be 0.

[0144] In some embodiments, the scale matrix may be constructed by referring to the following formula (5):

[0145]

[0146] In formula (5), M s represents the scale matrix and s represents the scale factor.

[0147] In step S0352, a rotation matrix is ​​constructed using the rotation factor, the preset first value and the preset second value.

[0148] Continuing with the above example, the preset first value may be 1, and the preset second value may be 0.

[0149] In some embodiments, the rotation matrix may be constructed with reference to the following formula (6):

[0150]

[0151] In formula (6), M r represents the rotation matrix, and Δθ represents the rotation factor.

[0152] In step S0353, a displacement matrix is ​​constructed using the displacement vector, the preset first value and the preset second value.

[0153] Continuing with the above example, the preset first value may be 1, and the preset second value may be 0.

[0154] In some embodiments, the displacement matrix may be constructed by referring to the following formula (7):

[0155]

[0156] In formula (7), M d represents the displacement matrix, and (Δx, Δy) represents the displacement vector.

[0157] In step S0354, the matrix product of the scale matrix, the rotation matrix and the displacement matrix is ​​determined as the transformation matrix.

[0158] In the embodiment of the present application, the matrix multiplication can be performed in the order of the scale matrix, the rotation matrix and the displacement matrix to obtain the transformation matrix. Refer to the following formula (8):

[0159] T current_keyframe_i =M s *M r *M d Formula (8);

[0160] In formula (8), T current_keyframe_i represents the transformation matrix, M s represents the scale matrix, M r represents the rotation matrix, M d represents the displacement matrix.

[0161] Through step S0351 to step S0354, the scale matrix, rotation matrix and displacement matrix are first constructed respectively, and then the transformation matrix is ​​determined.

[0162] Through steps S1031 to S1035, based on frequency domain image registration, the scale factor and the rotation factor are first determined; then the first ground texture image is scaled and rotated to obtain a fourth ground texture image; then, the displacement vector is determined based on the third frequency domain image of the fourth ground texture image; then, the scale matrix, the rotation matrix and the displacement matrix are respectively constructed, and then the transformation matrix is ​​determined, which can improve the efficiency of obtaining the transformation matrix and the accuracy of the transformation matrix.

[0163] Continue to see Figure 3 , and the description continues with step S103 above.

[0164] In step S104, first positioning information of the electronic device is determined based on the conversion matrix and the reference positioning information of the second ground texture image.

[0165] In the embodiment of the present application, referring to the above formulas (5) to (8), it can be seen that the conversion matrix is ​​a 3x3 matrix, and combined with the reference positioning information being a 3x1 matrix, the product of the conversion matrix and the reference positioning information is determined as the first positioning information of the electronic device. The first positioning information is a 3x1 matrix, wherein the first row matrix elements and the second row matrix elements represent the position information of the electronic device, and the third row matrix elements represent the angle of the electronic device compared to the reference direction when collecting the first ground texture image.

[0166] Through the above steps S101 to S104, the conversion matrix between the first ground texture image and the second ground image is determined based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, that is, the embodiment of the present application performs image registration based on the frequency domain space. Since the noise is concentrated in the high-frequency part of the frequency domain, and the illumination change is reflected in the low-frequency part of the frequency domain, the noise and illumination change can be filtered out during the image registration process, thereby improving the accuracy of image registration, and then improving the accuracy of positioning. In addition, the image to be registered in the embodiment of the present application is a ground texture image, rather than an image on the ground that is easily changed. Since the ground texture image has the characteristics of small frequency and small amplitude of change, it can improve the stability of positioning in dynamic scenes. In addition, since the electronic device can collect the first ground texture image in real time during the positioning process, and then perform positioning based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, the purpose of continuous positioning can be achieved.

[0167] In some embodiments, path tracking can also be performed based on the first positioning information of the electronic device. Figure 7 After the above step S104, the following steps S105 to S109 may also be executed, which are described in detail below.

[0168] In step S105, first movement information of the electronic device is collected.

[0169] In some embodiments, the first movement information may include the moving direction and moving distance of the electronic device, and a wheel speed sensor may be deployed on the electronic device, and the wheel speed odometer information of the electronic device may be collected through the wheel speed sensor, and the wheel speed odometer information may be considered as the first movement information. The wheel speed odometer information estimates the moving distance and direction of the vehicle or robot by measuring the rotation of the wheel.

[0170] In step S106, a posture prediction is performed based on the first movement information to obtain third positioning information and first posture information.

[0171] In some embodiments, the first historical position information of the electronic device at the last moment may be obtained, the first historical position information including the first historical positioning information and the first historical position information, and then the extended Kalman filter is used to make a prediction based on the first movement information and the first historical position information to obtain the third positioning information and the first position information. Among them, the position information refers to the orientation or direction of the electronic device relative to the reference coordinate system, and the position information may include the rotation angle of the electronic device in three-dimensional space.

[0172] In step S107, the first positioning information and the third positioning information are integrated to obtain fourth positioning information of the electronic device.

[0173] In some embodiments, the fusion process can be implemented through weighted processing, based on which a first weight of the first positioning information and a second weight of the third positioning information can be first obtained; then based on the first weight and the second weight, the first positioning information and the second positioning information are weightedly summed to obtain the fourth positioning information.

[0174] In step S108, the fourth positioning information and the first posture information are determined as the first posture information of the electronic device.

[0175] In some embodiments, the fourth positioning information and the first posture information may be combined to obtain the first posture information including the positioning information and the posture information.

[0176] In step S109, based on the first posture information and the preset moving path, the moving direction and the moving speed are determined; and the electronic device is controlled to move according to the moving direction and the moving speed.

[0177] In some embodiments, the preset moving path may be a pre-planned path, which is composed of a series of path points. The implementation method of determining the moving direction and moving speed includes: determining the next path point of the fourth positioning information from the preset moving path and using it as the target point; then determining the direction vector from the first posture information to the target point to obtain the moving direction; determining the moving speed according to the requirements of the preset path or the dynamic programming algorithm, which may be a constant speed or dynamically adjusted according to the path points and the environment; finally, controlling the electronic device to move according to the moving speed and along the moving direction.

[0178] Through the above steps S105 to S109, more accurate fourth positioning information is obtained by fusion, and accurate positioning of the electronic device is achieved. Furthermore, the moving direction and moving speed can be determined, and the electronic device can be controlled to move. In addition, the fourth positioning information of the electronic device can also be applied to navigation and path planning, task execution, human-computer interaction, multi-machine collaboration, etc.

[0179] In some other embodiments, before executing the above step S101, a texture map may be constructed. Figure 8 Before executing the above step S101, the following steps S001 to S005 may also be executed, which are described in detail below.

[0180] In step S001, a target area corresponding to a texture map to be constructed is determined.

[0181] In some embodiments, the target area may be an area set in advance, or may be an area determined by the electronic device based on obstacle information in the environment.

[0182] In step S002, the electronic device is controlled to move in the target area, and a fifth ground texture image in the target area is periodically collected.

[0183] In some embodiments, the electronic device may preset a time length as a period, and use the image acquisition module to periodically acquire the fifth ground texture image in the target area. Exemplarily, the preset time length may be 1 second, 2 seconds, 3 seconds, etc.

[0184] In step S003, fifth positioning information of the electronic device and second movement information of the electronic device are obtained, and position prediction is performed based on the second movement information to obtain sixth positioning information and second posture information.

[0185] In some embodiments, the fifth positioning information of the electronic device can be obtained by using the positioning module, and the second movement information can be collected by using the wheel speed sensor, and the second movement information includes the movement distance and the movement direction. The second historical posture information of the electronic device at the last moment can be obtained first, and the second historical posture information includes the second historical positioning information and the second historical posture information, and then the extended Kalman filter is used to predict based on the second movement information and the second historical posture information to obtain the sixth positioning information and the second posture information.

[0186] In step S004, the fifth positioning information and the sixth positioning information are fused to obtain seventh positioning information of the fifth ground texture image.

[0187] In some embodiments, the implementation process of step S004 is similar to the implementation process of step S107, and therefore, the implementation process of step S004 may refer to the implementation process of step S107.

[0188] In step S005, a texture map is constructed using each fifth ground texture image and each seventh positioning information.

[0189] In some embodiments, the fifth ground texture image and the corresponding seventh positioning information are determined as map key frames, based on which each fifth ground texture image and each seventh positioning information constitute each map key frame, thereby obtaining a texture map.

[0190] Through the above steps S001 to S005, a texture map including the fifth ground texture image and the seventh positioning information can be obtained. Since the texture map is associated with the ground texture, the constructed texture map can remain effective when the ground texture does not change, thereby improving the robustness of the texture map.

[0191] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0192] Based on the consideration of environmental changes, deployment costs and observation continuity, the embodiment of the present application proposes a method for positioning based on ground texture, which solves the impact of environmental changes on positioning and the high deployment cost of QR code positioning. The embodiment of the present application proposes a positioning method, which is a method for dynamic scene positioning based on ground texture and frequency domain matching. The positioning method provided by the embodiment of the present application can solve the impact of environmental changes on positioning, and can also avoid the problem of high deployment cost of QR code positioning, thereby achieving accurate positioning in dynamic scenes and reducing positioning costs.

[0193] The embodiment of the present application fully utilizes the characteristics of the three degrees of freedom of motion of the AGV robot, namely x, y and yaw. During the positioning process, image registration is performed based on frequency domain information, and the positioning method can achieve visual positioning of dynamic scenes without modifying the environment.

[0194] During the deployment process, the embodiment of the present application uses an extended Kalman filter (EKF) to fuse the global positioning information and wheel speed odometer information output by the previous module to obtain the global positioning information of each ground texture image, and save the ground texture image and the corresponding global positioning information as the map key frame. Because this is the deployment process, the previous laser SLAM module or visual SLAM module has just completed the mapping, and the environment has not changed, so it can be considered that the global positioning information output by the previous module is accurate.

[0195] In the positioning process, the scale transformation, rotation transformation and displacement transformation between the input image and the map key frame of the neighborhood search are determined by using Fourier Merllin Transform (FMT); then, the EKF is used to fuse the FMT results and the wheel speed odometer information to finally obtain the current real-time posture of the robot. Among them, the scale transformation corresponds to the scale factor in other embodiments, the rotation transformation corresponds to the rotation factor in other embodiments, the displacement transformation corresponds to the displacement vector in other embodiments, the current real-time posture corresponds to the first positioning information in other embodiments, and the map key frame of the neighborhood search corresponds to the second ground texture image in other embodiments. In some embodiments, the map key frame of the neighborhood search can be obtained by nearest neighbor search or radius search.

[0196] In some embodiments, the process of determining the current real-time posture is as follows:

[0197] (1) The AGV robot determines the frequency domain images of the input image and map key frames.

[0198] Here, the Fourier transform of the input image and the map key frame can be determined based on the following formula (9), thereby obtaining the frequency domain images of the input image and the map key frame. The input image is the ground texture image collected by the robot during the positioning process, corresponding to the first ground texture image in other embodiments; the map key frame refers to the map key frame of the neighborhood search, that is, the second ground texture image.

[0199]

[0200] In formula (9), u, v are the coordinates of the frequency domain image, F(u, v) is the Fourier transform of the time domain image, is the coordinate of the time domain image, f(x, y) represents the time domain image, and M and N are the length and width of the time domain image.

[0201] Through the above formula (9), the Fourier transform F1(u, v) of the input image and the Fourier transform F2(u, v) of the map key frame can be obtained respectively, wherein F1(u, v) is the frequency domain image of the input image, corresponding to the first frequency domain image in other embodiments; F2(u, v) is the frequency domain image of the map key frame, corresponding to the second frequency domain image in other embodiments.

[0202] (2) The AGV robot converts the frequency domain image F(u, v) after Fourier transformation into a logarithmic polar coordinate system (ρ, θ), and expresses the radial direction under the polar coordinates as the logarithmic radial direction, see the above formula (1); and expresses the angle under the polar coordinates as the angle shown in the above formula (2).

[0203] By transforming the above coordinate system, the scale change of the image is converted into a horizontal translation, see formula (10); and the rotation change of the image is converted into a vertical translation, see formula (11).

[0204]

[0205] Δθ=θ-θ, formula (11);

[0206] Here, the image obtained by transforming F1(u, v) by the coordinate system is recorded as F1(ρ, θ), and the image obtained by transforming F2(u, v) by the coordinate system is recorded as F2(ρ, θ).

[0207] (3) The AGV robot determines the phase correlation matrix of the frequency domain images in two logarithmic polar coordinate systems.

[0208] First, perform Fourier transform based on formula (12) to obtain G(ρ, θ) and H(ρ, θ); then determine the conjugate phase product R(ρ, θ) based on formula (13); finally, determine the phase correlation matrix r(x, y) based on formula (14).

[0209]

[0210] Among them, in formula (12) represents Fourier transform, H in formula (13) * (ρ, θ) is the conjugate complex number of H(ρ, θ), and the represents the inverse Fourier transform.

[0211] (4) The AGV robot determines the scale transformation and rotation transformation of the two images through the peak position (Δρ, Δθ) of the phase correlation matrix r(x, y), wherein the scale transformation can be determined by the above formula (3), that is, the scale factor is determined; the rotation transformation can be determined by the above formula (4), that is, the rotation factor is determined.

[0212] (5) The AGV robot determines the input image after rotation.

[0213] In some embodiments, the scale-transformed image can be scaled by the following formula (15) to obtain a scale-transformed image; and the scale-transformed image can be rotated by the following formula (16) to obtain a rotated input image. The scale-transformed image corresponds to the third ground texture image in other embodiments, and the rotated input image corresponds to the fourth ground texture image in other embodiments.

[0214] f s (x, y)=f(x / s, y / s) formula (15);

[0215] fr (x, y) = f s (x cosΔθ-y sinΔθ, x sinΔθ+y cosΔθ) formula (16);

[0216] Among them, f s (x, y) represents the image after scale transformation, f r (x, y) represents the input image after rotation.

[0217] (6) The AGV robot determines the Fourier transform of the rotated input image and the map key frame image in the Cartesian coordinate system.

[0218] In some embodiments, the Fourier transform of the rotated input image in the Cartesian coordinate system can be determined by the following formula (17), denoted as F r (u, v). In addition, in the above (1), the Fourier transform F2 (u, v) of the map key frame in the Cartesian coordinate system has been determined.

[0219]

[0220] (7) The AGV robot determines the phase correlation matrix of the frequency domain image in two Cartesian coordinate systems.

[0221] In the embodiment of the present application, the conjugate phase product can be determined by the following formula (18); and then the phase correlation matrix can be determined by the following formula (19), which refers to the phase correlation matrix between the rotated input image and the map key frame image.

[0222]

[0223] In formula (18), F2 * (u, v) is the conjugate complex number of F2(u, v), and R(u, v) represents the conjugate phase product; in formula (19), represents the inverse Fourier transform, and r(x, y) represents the phase correlation matrix.

[0224] (8) The AGV robot determines the displacement transformation of the two images through the peak position (Δx, Δy) of the phase correlation matrix r(x, y).

[0225] In the embodiment of the present application, the displacement transformation refers to the displacement transformation between the input image and the map key frame image. The formula for determining the displacement transformation can be found in formula (20).

[0226] (Δx, Δy) = argmax(r(x, y)) Formula (20);

[0227] In this way, the scale transformation, rotation transformation and displacement transformation of the input image and the map key frame are obtained. Based on this, the transformation matrix can be constructed based on formula (21).

[0228]

[0229] Among them, T current_keyframe_i represents the transformation matrix, s represents the scale transformation (scale factor), Δθ represents the rotation transformation (rotation factor), and (Δx, Δy) represents the displacement transformation (displacement vector).

[0230] In some embodiments, because the global positioning information T of the map keyframe keyframe_i It has been obtained during the mapping process, so the global positioning information of the input image can be obtained by the following formula (22). Here, the global positioning information of the input image corresponds to the first positioning information in other embodiments.

[0231] T current =T current_keyframe_i *T keyframe_i Formula (22);

[0232] In formula (22), it represents the transformation matrix, T keyframe_i Represents the global positioning information of the map keyframe, T current Represents the global positioning information of the input image.

[0233] Finally, the extended Kalman filter is used to fuse the global positioning information and wheel speed odometer information of the input image to obtain the current real-time posture of the robot, where the current real-time posture corresponds to the first posture information in other embodiments.

[0234] Since the images to be registered are images of ground textures, rather than information such as feature points in the ground environment, the change amplitude and frequency of ground textures are relatively small compared to the easily changing ground environment, so it can have higher long-term positioning performance in dynamic scenes. Compared with ground QR codes, ground textures do not require complex manual deployment, which greatly reduces deployment costs, and ground textures can provide continuous observation information to the system compared to QR codes. In the process of texture map construction, the positioning information output by the preceding module is used as the global positioning information of the map key frame, so this function can be added to the existing AGV positioning system. In addition, since the embodiment of the present application is based on frequency domain space for image registration, since the noise is concentrated in the high-frequency part of the frequency domain, the positioning method provided by the embodiment of the present application is more robust to noise in the image; image registration can be completed through a phase correlation matrix to improve positioning efficiency; since the illumination changes are mainly reflected in the low-frequency part, the frequency domain method can avoid the influence of noise on image registration, thereby improving illumination robustness.

[0235] It can be understood that in the embodiments of the present application, related data such as the first ground texture image, the pre-built texture map, the first movement information, the preset movement path, etc. are involved, and the collection, use and processing of the related data need to comply with relevant laws, regulations and standards.

[0236] The following is a description of an exemplary structure of the positioning device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the positioning device 455 of the memory 450 may include:

[0237] The first acquisition module 4551 is used to acquire a first ground texture image and obtain a pre-built texture map, wherein the texture map includes a plurality of reference ground texture images and reference positioning information of each reference ground texture image;

[0238] A first determining module 4552 is used to determine a second ground texture image that satisfies a position relationship condition with the first ground texture image from the plurality of reference ground texture images;

[0239] A second determining module 4553 is configured to determine a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image;

[0240] The third determination module 4554 is used to determine the first positioning information of the electronic device based on the conversion matrix and the reference positioning information of the second ground texture image.

[0241] In some embodiments, the first determining module 4552 is further configured to:

[0242] Based on the first time domain image of the first ground texture image, second positioning information is determined; based on the second positioning information and the reference positioning information of each of the reference ground texture images, a first distance between a first position corresponding to the first ground texture image and a reference position corresponding to each of the reference ground texture images is determined; the reference ground texture image corresponding to the first distance less than the distance threshold is determined as a second ground texture image that satisfies a positional relationship condition with the first ground texture image; alternatively, the first distances are sorted in ascending order to obtain sorted first distances; and the reference ground texture images corresponding to the first N first distances in the sorted first distances are determined as second ground texture images that satisfy a positional relationship condition with the first ground texture image, where N is a positive integer.

[0243] In some embodiments, the second determining module 4553 is further configured to:

[0244] Determining a scale factor and a rotation factor based on the first frequency domain image and the second frequency domain image;

[0245] The first ground texture image is scaled using the scale factor to obtain a third ground texture image; the third ground texture image is rotationally transformed using the rotation factor to obtain a fourth ground texture image; a displacement vector is determined based on the third frequency domain image and the second frequency domain image of the fourth ground texture image; and the transformation matrix is ​​determined based on the scale factor, the rotation factor and the displacement vector.

[0246] In some embodiments, the second determining module 4553 is further configured to:

[0247] Perform coordinate transformation on the first frequency domain image to obtain a fourth frequency domain image of the first ground texture image in a log polar coordinate system; perform coordinate transformation on the second frequency domain image to obtain a fifth frequency domain image of the third ground texture image in the log polar coordinate system; determine a first phase correlation matrix between the first ground texture image and the third ground texture image based on the fourth frequency domain image and the fifth frequency domain image; determine the scale factor and the rotation factor based on the peak position of the first phase correlation matrix.

[0248] In some embodiments, the second determining module 4553 is further configured to:

[0249] Based on the third frequency domain image and the second frequency domain image, a second phase correlation matrix between the fourth ground texture image and the third ground texture image is determined; based on the peak position of the second phase correlation matrix, the displacement vector is determined.

[0250] In some embodiments, the second determining module 4553 is further configured to:

[0251] A scale matrix is ​​constructed using the scale factor, the preset first value and the preset second value; a rotation matrix is ​​constructed using the rotation factor, the preset first value and the preset second value; a displacement matrix is ​​constructed using the displacement vector, the preset first value and the preset second value; and the matrix product of the scale matrix, the rotation matrix and the displacement matrix is ​​determined as the transformation matrix.

[0252] In some embodiments, the software modules stored in the positioning device 455 of the memory 450 may also include:

[0253] A second collection module, used for collecting first movement information of the electronic device;

[0254] A first prediction module, configured to perform posture prediction based on the first movement information to obtain third positioning information and first posture information;

[0255] A first fusion module, used to fuse the first positioning information and the third positioning information to obtain fourth positioning information of the electronic device;

[0256] a fourth determining module, configured to determine the fourth positioning information and the first posture information as the first posture information of the electronic device;

[0257] The fifth determination module is used to determine the moving direction and the moving speed based on the first posture information and the preset moving path; and control the electronic device to move according to the moving direction and the moving speed.

[0258] In some embodiments, the software modules stored in the positioning device 455 of the memory 450 may also include:

[0259] A sixth determination module, used to determine a target area corresponding to the texture map to be constructed;

[0260] A third acquisition module, used for controlling the electronic device to move in the target area and periodically acquiring a fifth ground texture image in the target area;

[0261] an acquisition module, configured to acquire fifth positioning information of the electronic device and second movement information of the electronic device, and perform posture prediction based on the second movement information to obtain sixth positioning information and second posture information;

[0262] A second fusion module, used for fusing the fifth positioning information and the sixth positioning information to obtain seventh positioning information of the fifth ground texture image;

[0263] A construction module is used to construct the texture map by using each of the fifth ground texture images and each of the seventh positioning information.

[0264] The embodiment of the present application provides a computer program product or a computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the positioning method described in the embodiment of the present application.

[0265] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the positioning method provided by the embodiment of the present application, for example, Figure 3 , Figure 7 , Figure 8 The positioning method is shown.

[0266] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0267] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0268] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0269] By way of example, computer executable instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0270] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A positioning method, characterized in that: The method comprises: Collecting a first ground texture image and obtaining a pre-built texture map, wherein the texture map includes a plurality of reference ground texture images and reference positioning information of each of the reference ground texture images; Determine, from the plurality of reference ground texture images, a second ground texture image that satisfies a position relationship condition with the first ground texture image; determining a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image; Based on the conversion matrix and the reference positioning information of the second ground texture image, first positioning information of the electronic device is determined.

2. The method according to claim 1, characterized in that The step of determining, from the plurality of reference ground texture images, a second ground texture image that satisfies a position relationship condition with the first ground texture image comprises: Determining second positioning information based on the first time domain image of the first ground texture image; Determine, based on the second positioning information and the reference positioning information of each of the reference ground texture images, a first distance between a first position corresponding to the first ground texture image and a reference position corresponding to each of the reference ground texture images; Determine a reference ground texture image corresponding to a first distance less than a distance threshold as a second ground texture image that satisfies a position relationship condition with the first ground texture image; or The first distances are sorted in ascending order to obtain sorted first distances; and the reference ground texture images corresponding to the first N first distances in the sorted first distances are determined as second ground texture images that meet a positional relationship condition with the first ground texture image, where N is a positive integer.

3. The method according to claim 1, characterized in that The determining, based on the first frequency domain image of the first ground texture image and the second frequency domain image of the second ground texture image, a conversion matrix between the first ground texture image and the second ground texture image comprises: Determining a scale factor and a rotation factor based on the first frequency domain image and the second frequency domain image; Using the scale factor to scale the first ground texture image to obtain a third ground texture image; Performing a rotation transformation on the third ground texture image using the rotation factor to obtain a fourth ground texture image; determining a displacement vector based on the third frequency domain image of the fourth ground texture image and the second frequency domain image; The transformation matrix is ​​determined based on the scale factor, the rotation factor, and the displacement vector.

4. The method according to claim 3, characterized in that The determining of a scale factor and a rotation factor based on the first frequency domain image and the second frequency domain image includes: Performing coordinate transformation on the first frequency domain image to obtain a fourth frequency domain image of the first ground texture image in a logarithmic polar coordinate system; Performing coordinate transformation on the second frequency domain image to obtain a fifth frequency domain image of the third ground texture image in the logarithmic polar coordinate system; Determine a first phase correlation matrix between the first ground texture image and the third ground texture image based on the fourth frequency domain image and the fifth frequency domain image; The scale factor and the rotation factor are determined based on a peak position of the first phase correlation matrix.

5. The method according to claim 4, characterized in that The determining of the displacement vector based on the third frequency domain image of the fourth ground texture image and the second frequency domain image comprises: Determine a second phase correlation matrix between the fourth ground texture image and the third ground texture image based on the third frequency domain image and the second frequency domain image; The displacement vector is determined based on the peak position of the second phase correlation matrix.

6. The method according to claim 3, characterized in that The determining the transformation matrix based on the scale factor, the rotation factor and the displacement vector comprises: constructing a scale matrix using the scale factor, the preset first value and the preset second value; Constructing a rotation matrix using the rotation factor, the preset first value and the preset second value; constructing a displacement matrix using the displacement vector, the preset first value and the preset second value; The transformation matrix is ​​determined by multiplying the scale matrix, the rotation matrix and the displacement matrix.

7. The method according to any one of claims 1 to 6, characterized in that: After determining the first positioning information of the electronic device, the method further includes: Collecting first movement information of the electronic device; Performing posture prediction based on the first movement information to obtain third positioning information and first posture information; fusing the first positioning information and the third positioning information to obtain fourth positioning information of the electronic device; Determine the fourth positioning information and the first posture information as the first posture information of the electronic device; Based on the first posture information and the preset moving path, a moving direction and a moving speed are determined; and the electronic device is controlled to move according to the moving direction and the moving speed.

8. The method according to any one of claims 1 to 6, characterized in that: Before obtaining the pre-built texture map, the method further includes: Determine the target area corresponding to the texture map to be constructed; Controlling the electronic device to move in the target area, and periodically collecting a fifth ground texture image in the target area; Acquire fifth positioning information of the electronic device and second movement information of the electronic device, perform posture prediction based on the second movement information, and obtain sixth positioning information and second posture information; fusing the fifth positioning information and the sixth positioning information to obtain seventh positioning information of the fifth ground texture image; The texture map is constructed by using each of the fifth ground texture images and each of the seventh positioning information.

9. A positioning device, characterized in that: The device comprises: A first acquisition module is used to acquire a first ground texture image and obtain a pre-built texture map, wherein the texture map includes a plurality of reference ground texture images and reference positioning information of each of the reference ground texture images; A first determining module, configured to determine, from the plurality of reference ground texture images, a second ground texture image that satisfies a position relationship condition with the first ground texture image; A second determining module, configured to determine a conversion matrix between the first ground texture image and the second ground texture image based on a first frequency domain image of the first ground texture image and a second frequency domain image of the second ground texture image; The third determination module is used to determine the first positioning information of the electronic device based on the conversion matrix and the reference positioning information of the second ground texture image.

10. A computer device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, used to implement the positioning method described in any one of claims 1 to 8 when executing the computer executable instructions stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the positioning method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the positioning method according to any one of claims 1 to 8 is implemented.

Images