Indoor positioning method, system and device based on cross-modal fusion and medium

Abstract

The invention relates to the technical field of cross-modal fusion, and discloses an indoor positioning method, system and device based on cross-modal fusion and a medium, and the method comprises the steps: transmitting a positioning signal source to a plurality of sensors through a wireless transmission mode; carrying out feature extraction on the positioning signal source data received by the plurality of sensors by adopting a cross-modal factorization Hash mode and realizing common space mapping to generate Hash codes of the plurality of sensors; associating the hash codes of the plurality of sensors, and aligning the hash codes of the plurality of sensors; and introducing an attention mechanism to calculate the confidence of the aligned Hash codes, and carrying out feature fusion based on the confidence to realize position prediction. Cross-modal alignment is carried out in a CCA maximization mode, and alignment of positioning data of positioning frequencies and positioning precision of different information sources is achieved; an attention mechanism is introduced to calculate the feature confidence coefficient of the multi-modal data, multi-modal feature fusion is carried out, position prediction is obtained, and indoor fusion positioning of the non-homologous asynchronous data is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal fusion, and in particular to an indoor positioning method, system, device and medium based on cross-modal fusion. Background Art

[0002] In recent years, the rapid development of intelligent robots has greatly changed the way people live. Intelligent robots can replace humans in complex indoor environments, such as mine tunnels, nuclear waste disposal sites, high-temperature flammable and explosive sites, and toxic gas emission sites, and realize autonomous navigation and control of robots in complex environments. However, the autonomous navigation and control of intelligent robots includes modules such as perception and positioning, route planning, and autonomous decision-making and control. Among them, autonomous positioning is the prerequisite for autonomous navigation and control of intelligent robots. At present, most of the indoor positioning methods of intelligent robots use multiple information fusion methods. Most of these methods directly use data from multiple sensors to perform feature fusion and then realize position estimation.

[0003] The existing technology proposes a hierarchical indoor positioning method that integrates Wi-Fi, magnetic matching and pedestrian dead reckoning. After Wi-Fi and magnetic matching form an absolute positioning result, an adaptive extended Kalman filter is used to integrate pedestrian dead reckoning and absolute positioning results, thereby utilizing the advantages of various information sources to form better positioning.

[0004] The existing technology of indoor navigation intelligent logistics precise positioning using ultra-wideband technology is reviewed, and the advantages of ultra-wideband fusing with other sensors to achieve precise positioning and target tracking are introduced.

[0005] The existing technology proposes an indoor positioning algorithm based on multi-source data fusion, proposes the fusion positioning of Wi-Fi and RFID, and a multi-source positioning model based on an inertial measurement unit and introduces a self-correcting particle swarm optimization algorithm to realize position calculation. Compared with the traditional algorithm, the maximum positioning error of this method is reduced by about 150%.

[0006] The prior art proposes a magnetic indoor positioning algorithm for smartphones based on a fusion algorithm. The algorithm adopts an extended Kalman filter and a particle filter method to implement a two-stage positioning architecture from coarse to fine, thereby improving the speed and accuracy of positioning.

[0007] The prior art proposes a multi-source information fusion indoor positioning method based on genetic algorithm optimized neural network. This method uses the Wi-Fi / geomagnetic combined positioning results and the pedestrian dead reckoning positioning results as neural network inputs, uses the BP neural network, and uses the real position as the target output to obtain a fusion positioning result. However, sensors are easily affected by complex environments due to their own reasons. For example, some devices will fail in high temperature and high electromagnetic environments. In addition, due to inconsistent collection frequencies between devices, data asynchrony is easily caused. Finally, multiple sensor data will generate a large amount of redundant data, which will affect the positioning delay. Summary of the invention

[0008] The present invention provides an indoor positioning method, system, device and medium based on cross-modal fusion, which adopts the CCA maximization method to achieve cross-modal alignment of multimodal data, and realizes the alignment of positioning data of positioning frequency and positioning accuracy of different sources; introduces an attention mechanism to calculate the confidence of multimodal data features, and performs multimodal feature fusion based on the confidence to obtain position prediction, thereby realizing the fusion positioning problem of non-homologous asynchronous data in indoor areas.

[0009] The present invention provides an indoor positioning method based on cross-modal fusion, comprising:

[0010] Transmitting the positioning signal source to a plurality of sensors by wireless transmission;

[0011] The cross-modal factor decomposition hashing method is used to extract features from the positioning signal source data received by several sensors and realize common space mapping, and the hash codes of several sensors are generated in space;

[0012] Associating hash codes of several sensors and aligning hash codes of several sensors;

[0013] The attention mechanism is introduced to calculate the confidence of the aligned hash codes, and feature fusion is performed based on the confidence to achieve position prediction.

[0014] Preferably, the wireless transmission method includes Bluetooth and WIFI.

[0015] Preferably, the plurality of sensors include an odometer, an accelerometer, a direction sensor and a gyroscope.

[0016] Preferably, the cross-modal factorization hashing method includes:

[0017] The data dimensions of several sensors are obtained, and the common potential features of several sensors are learned by cross-modal factor decomposition hashing combined with position semantic tags. The multimodal data hash features are extracted using the minimum loss function form.

[0018] Preferably, associating the hash codes of the plurality of sensors and aligning the hash codes of the plurality of sensors comprises:

[0019] After learning the feature hash codes of several sensors, the hash codes of several sensors are associated, and the cross-modal alignment of the hash codes is achieved by maximizing CCA.

[0020] Preferably, the introduction of the attention mechanism to calculate the confidence of the aligned hash code and the feature fusion based on the confidence to achieve the position prediction includes:

[0021] Build a training model; the training model includes input layer, encoding layer, embedding layer, feature fusion layer and fully connected layer;

[0022] The hash codes of several sensors are input into the input layer of the training model, and the hash codes are encoded in the encoding layer to obtain the encoded representation and then input into the embedding layer;

[0023] The embedding layer introduces an attention mechanism to obtain the embedding representation of the hash codes of several sensors and calculate the hash code confidence of several sensors;

[0024] The feature fusion layer fuses the hash code confidence and encoding representation of several sensors and inputs them into the fully connected layer to predict the location information.

[0025] An indoor positioning system based on cross-modal fusion, comprising:

[0026] A signal source module, used for transmitting the positioning signal source to a plurality of sensors by wireless transmission;

[0027] A spatial mapping module is used to extract features of positioning signal source data received by a plurality of sensors and implement common spatial mapping by adopting a cross-modal factor decomposition hashing method, and generate hash codes of a plurality of sensors in space;

[0028] A hash code alignment module, used for associating hash codes of a plurality of sensors and aligning the hash codes of the plurality of sensors;

[0029] The feature fusion module is used to introduce the attention mechanism to calculate the confidence of the aligned hash codes, and perform feature fusion based on the confidence to achieve position prediction.

[0030] A terminal device comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements an indoor positioning method based on cross-modal fusion as described when executing the computer program.

[0031] A computer-readable storage medium, comprising a stored computer program; wherein, when the computer program is run, the device where the computer-readable storage medium is located is controlled to execute an indoor positioning method based on cross-modal fusion as described above.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention discloses an indoor positioning method, system, device and medium based on cross-modal fusion, which adopts a CCA maximization method to achieve cross-modal alignment of multimodal data, and realizes the alignment of positioning data of positioning frequency and positioning accuracy of different sources; introduces an attention mechanism to calculate the confidence of multimodal data features, and performs multimodal feature fusion based on the confidence to obtain position prediction, thereby realizing the fusion positioning problem of non-homologous asynchronous data in indoor areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of a flow chart of an indoor positioning method based on cross-modal fusion provided by an embodiment of the present invention;

[0035] Figure 2 is a flow chart of realizing position prediction through feature fusion provided by an embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of an indoor positioning system module based on cross-modal fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, the present application provides an indoor positioning method based on cross-modal fusion, including:

[0039] S1: Transmit the positioning signal source to several sensors by wireless transmission;

[0040] S2: Use cross-modal factorization hashing to decompose the data received by several sensors and realize common space mapping, and generate hash codes of several sensors in space;

[0041] S3: Associating hash codes of several sensors and aligning hash codes of several sensors;

[0042] S4: The attention mechanism is introduced to calculate the confidence of the aligned hash codes, and feature fusion is performed based on the confidence to achieve position prediction.

[0043] In the above scheme, the positioning frequency and positioning accuracy of different sources have certain differences. Based on common spatial mapping and feature extraction, this application adopts CCA maximization to achieve cross-modal alignment of multimodal data, which can achieve alignment of positioning data to a certain extent and lay the foundation for subsequent fusion of positioning data features; in the fusion of positioning data, this application introduces an attention mechanism to calculate the confidence of multimodal data features, and performs multimodal feature fusion based on the confidence to obtain position prediction; this application solves the problems of common spatial mapping and feature extraction of multimodal data, cross-modal alignment of multimodal data, and confidence of multimodal data features, thereby realizing the fusion positioning problem of non-homologous asynchronous data in indoor areas.

[0044] Preferably, in step S1, the wireless transmission method includes Bluetooth and WIFI; the plurality of sensors include an odometer, an accelerometer, a direction sensor and a gyroscope.

[0045] Preferably, in step S2, the cross-modal factor decomposition hashing method includes:

[0046] The data dimensions of several sensors are obtained, and the common potential features of several sensors are learned by cross-modal factor decomposition hashing combined with position semantic tags. The multimodal data hash features are extracted using the minimum loss function form.

[0047] In the above scheme, cross-modal factorization hashing combines the methods of matrix decomposition and hash learning. By jointly decomposing data of different modalities, a shared low-dimensional feature space is learned to achieve common space mapping and generate hash codes in this space.

[0048] In one embodiment provided in this application, there are two types of sensor data: and Among them, d1 and d2 represent the data dimensions of a single sample obtained by the first type of sensor, and n represents the number of samples. The cross-modal factorization hashing is combined with the position semantic label L to learn the common potential features of two different sensors, and the multimodal data hash feature extraction is realized in the form of minimizing the loss function:

[0049]

[0050] Among them, W is the regression coefficient matrix, which is continuously optimized with sample iteration; L is the position semantic label. If the positioning of the two sensors is consistent within a fixed observation period, such as 1 minute as 1 observation period, then L = 1, otherwise L = 0; matrices U and V are linear transformation matrices, through which the original image features and text features can be transformed into a unified latent semantic space; B is a hash matrix, and the elements in the hash matrix B are only 1 or -1; is the hash code corresponding to a sample of the first type of sensor; is the hash code corresponding to a sample of the second type of sensor; λ1 and λ2 are balance parameters with values ​​between 0 and 1.

[0051] The first item uses semantic label information to learn hash codes for positioning data of two types of sensors. If the positioning information of the two types of sensors is consistent, the element corresponding to the hash code is 1, otherwise it is -1. The hash codes learned from multiple samples of the two types of sensors have accurate identification. The second item is to learn a unified latent semantic space and data feature hash codes for the sample data of each type of sensor by minimizing the residual constraint, thereby realizing feature hash code extraction.

[0052] Preferably, in step S3, associating the hash codes of the plurality of sensors and aligning the hash codes of the plurality of sensors comprises:

[0053] After learning the feature hash codes of several sensors, the hash codes of several sensors are associated, and the cross-modal alignment of the hash codes is achieved by maximizing CCA.

[0054] In the above scheme, based on the feature hash code learning of sample data of two types of sensors, the hash codes of the two types of sensor data are usually associated to achieve cross-modal alignment by maximizing the correlation. This patent uses CCA to achieve alignment of multimodal data.

[0055] In an embodiment provided in the present application, the hash code alignment of data features is implemented using data from two types of sensors for illustration. Based on the hash code selection of the data features of the two types of sensors, the cross-modal alignment of multimodal data is implemented by maximizing CCA, as follows:

[0056]

[0057] in, represents the covariance of the hash signatures of the two types of sensors, and Respectively represent the variance of the hash feature codes of the two types of sensors. By maximizing the correlation between the two types of data features, cross-modal alignment of multimodal data can be achieved.

[0058] Preferably, in step S4, the introducing of the attention mechanism to calculate the confidence of the aligned hash code and performing feature fusion based on the confidence to achieve position prediction includes:

[0059] Build a training model; the training model includes input layer, encoding layer, embedding layer, feature fusion layer and fully connected layer;

[0060] The hash codes of several sensors are input into the input layer of the training model, and the hash codes are encoded in the encoding layer to obtain the encoded representation and then input into the embedding layer;

[0061] The embedding layer introduces an attention mechanism to obtain the embedding representation of the hash codes of several sensors and calculate the hash code confidence of several sensors;

[0062] The feature fusion layer fuses the hash code confidence and encoding representation of several sensors and inputs them into the fully connected layer to predict the location information.

[0063] like Figure 2 As shown, in one embodiment provided in the present application, after cross-modal alignment, the aligned multi-modal data feature hash codes are placed in an end-to-end training framework, and the multimodal data feature hash codes are encoded and placed in an encoding layer; the embedding layer introduces an attention mechanism to obtain a high-dimensional vector representation of each modality and calculates the confidence of the multimodal data feature hash code; the multimodal data feature confidence and the encoded representation are used to realize multimodal feature fusion, which is input into the fully connected layer to realize the prediction of position information.

[0064] like Figure 3 As shown, an indoor positioning system based on cross-modal fusion includes:

[0065] A signal source module, used for transmitting the positioning signal source to a plurality of sensors by wireless transmission;

[0066] A spatial mapping module, used to perform feature decomposition on data received by a plurality of sensors and implement common spatial mapping by adopting a cross-modal factor decomposition hashing method, and generate hash codes of a plurality of sensors in space;

[0067] A hash code alignment module, used for associating hash codes of a plurality of sensors and aligning the hash codes of the plurality of sensors;

[0068] The feature fusion module is used to introduce the attention mechanism to calculate the confidence of the aligned hash codes, and perform feature fusion based on the confidence to achieve position prediction.

[0069] In one embodiment provided in the present application, the positioning system is composed of a positioning signal source, a mobile terminal and a server. The positioning signal source includes a Bluetooth beacon installed indoors and a sensor built into the mobile terminal, and the sensor includes an odometer, an accelerometer, a direction sensor, a gyroscope, etc.

[0070] At time t, Bluetooth sends an ID to the mobile terminal device in the form of broadcast. The terminal expresses the received ID and signal strength as {ID, RSS} t , through the built-in sensor k, the positioning result at time t is The positioning frequency of each sensor is not completely consistent with the positioning frequency of Bluetooth. Therefore, after the positioning signal source data enters the edge server, it goes through feature extraction, common space mapping, cross-modal alignment, joint learning and other steps. Considering that multiple sensor data will generate a large amount of redundant data, hash codes are introduced in the process of common space mapping to achieve feature reconstruction.

[0071] A device comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements a project schedule prediction method based on deep learning when executing the computer program.

[0072] A computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the project progress prediction method based on deep learning.

[0073] Preferably, the computer program can be divided into one or more modules / units (such as computer program, computer program), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0074] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0075] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.

[0076] It should be noted that the above-mentioned terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above-mentioned terminal device is merely an example and does not constitute a limitation on the terminal device. It may include more or fewer components, or a combination of certain components, or different components.

[0077] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An indoor positioning method based on cross-modal fusion, characterized in that: include: Transmitting the positioning signal source to a plurality of sensors by wireless transmission; The cross-modal factor decomposition hashing method is used to extract features from the positioning signal source data received by several sensors and realize common space mapping, and the hash codes of several sensors are generated in space; Associating hash codes of several sensors and aligning hash codes of several sensors; The attention mechanism is introduced to calculate the confidence of the aligned hash codes, and feature fusion is performed based on the confidence to achieve position prediction.

2. According to claim 1, the indoor positioning method based on cross-modal fusion is characterized in that: The wireless transmission methods include Bluetooth and WIFI.

3. The indoor positioning method based on cross-modal fusion according to claim 1, characterized in that: Several sensors include an odometer, accelerometer, orientation sensor, and gyroscope.

4. The indoor positioning method based on cross-modal fusion according to claim 1, characterized in that: The cross-modal factorization hashing method includes: The data dimensions of several sensors are obtained, and the common potential features of several sensors are learned by cross-modal factor decomposition hashing combined with position semantic tags. The multimodal data hash features are extracted using the minimum loss function form.

5. The indoor positioning method based on cross-modal fusion according to claim 4, characterized in that: The associating the hash codes of the plurality of sensors and aligning the hash codes of the plurality of sensors comprises: After learning the feature hash codes of several sensors, the hash codes of several sensors are associated, and the cross-modal alignment of the hash codes is achieved by maximizing CCA.

6. The indoor positioning method based on cross-modal fusion according to claim 5, characterized in that: The attention mechanism is introduced to calculate the confidence of the aligned hash code, and feature fusion is performed based on the confidence to achieve position prediction, including: Build a training model; the training model includes input layer, encoding layer, embedding layer, feature fusion layer and fully connected layer; The hash codes of several sensors are input into the input layer of the training model, and the hash codes are encoded in the encoding layer to obtain the encoded representation and then input into the embedding layer; The embedding layer introduces an attention mechanism to obtain the embedding representation of the hash codes of several sensors and calculate the hash code confidence of several sensors; The feature fusion layer fuses the hash code confidence and encoding representation of several sensors and inputs them into the fully connected layer to predict the location information.

7. An indoor positioning system based on cross-modal fusion, characterized in that: include: A signal source module, used for transmitting the positioning signal source to a plurality of sensors by wireless transmission; A spatial mapping module is used to extract features of positioning signal source data received by a plurality of sensors and implement common spatial mapping by adopting a cross-modal factor decomposition hashing method, and generate hash codes of a plurality of sensors in space; A hash code alignment module, used for associating hash codes of a plurality of sensors and aligning the hash codes of the plurality of sensors; The feature fusion module is used to introduce the attention mechanism to calculate the confidence of the aligned hash codes, and perform feature fusion based on the confidence to achieve position prediction.

8. A terminal device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements an indoor positioning method based on cross-modal fusion as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute an indoor positioning method based on cross-modal fusion as described in any one of claims 1-6.

Images