Terminal equipment positioning method and device, equipment, storage medium and program product

Abstract

The invention discloses a terminal equipment positioning method and device, equipment, a storage medium and a program product, and relates to the technical field of positioning. The positioning method of the terminal equipment comprises the steps that a first positioning point and a plurality of second positioning points of the terminal equipment and multi-modal scene information of the terminal equipment are acquired, the first positioning point is a positioning point obtained through a target positioning mode, and the second positioning points are positioning points obtained through candidate positioning modes except the target positioning mode; according to the multi-mode scene information of the terminal equipment, determining a target scene type of the terminal equipment; determining a target second positioning point matched with the target scene type from the second positioning points according to the target scene type; according to the distance and the direction between the first positioning point and each target second positioning point, determining a matching positioning point with the highest matching degree with the first positioning point in each target second positioning point; and correcting the first positioning point through the matched positioning point to obtain the positioning information of the terminal equipment.

Description

Technical Field

[0001] The present application belongs to the field of positioning technology, and in particular, relates to a positioning method, device, equipment, storage medium and program product for a terminal device. Background Art

[0002] With the development of science and technology, mobile communication has become quite popular. In recent years, with the increase of navigation demand, the positioning technology of terminal equipment has developed rapidly.

[0003] In existing methods, terminal devices are usually positioned based on a single positioning method such as WIFI positioning to provide reliable navigation.

[0004] However, in complex scenarios or when the positioning algorithm fails, the positioning accuracy of a single positioning method such as WIFI positioning is low. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device, storage medium and program product for positioning a terminal device, which can improve the accuracy of positioning.

[0006] In one aspect of an embodiment of the present application, a method for positioning a terminal device is provided, the method comprising:

[0007] Acquire a first positioning point of a terminal device, multiple second positioning points, and multimodal scene information of a location of the terminal device, wherein the first positioning point is a positioning point obtained by a target positioning method, and the second positioning point is a positioning point obtained by a candidate positioning method other than the target positioning method;

[0008] Determine the target scene type of the terminal device according to the multimodal scene information corresponding to the terminal device;

[0009] According to the target scene type, determining a target second positioning point matching the target scene type from each second positioning point;

[0010] Determine, according to the distance and direction between the first positioning point and the second positioning point of each target, a matching positioning point with the highest matching degree with the first positioning point among the second positioning points of each target;

[0011] By matching the positioning points, the first positioning point is corrected to obtain the positioning information of the terminal device.

[0012] In one aspect of an embodiment of the present application, a positioning device for a terminal device is provided, the device comprising:

[0013] An information acquisition module, used to acquire a first positioning point of a terminal device, multiple second positioning points, and multimodal scene information of the location of the terminal device, wherein the first positioning point is a positioning point obtained by a target positioning method, and the second positioning point is a positioning point obtained by a candidate positioning method other than the target positioning method;

[0014] A type determination module, used to determine the target scene type of the terminal device according to the multimodal scene information corresponding to the terminal device;

[0015] A positioning point matching module, used to determine a target second positioning point matching the target scene type from each second positioning point according to the target scene type;

[0016] The positioning point matching module is further used to determine the matching positioning point with the highest matching degree with the first positioning point among the second positioning points of each target according to the distance and direction between the first positioning point and the second positioning point of each target;

[0017] The positioning correction module is used to correct the first positioning point by matching the positioning points to obtain the positioning information of the terminal device.

[0018] In one aspect of an embodiment of the present application, an electronic device is provided, which includes: a memory and a program or instruction stored in the memory and executable on a processor, wherein when the program or instruction is executed by the processor, a positioning method for a terminal device as provided in any aspect of the above-mentioned embodiment of the present application is implemented.

[0019] In one aspect of an embodiment of the present application, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, a positioning method for a terminal device as provided in any aspect of the above-mentioned embodiment of the present application is implemented.

[0020] In one aspect of an embodiment of the present application, a computer program product is provided. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes a positioning method for a terminal device provided in any aspect of the above-mentioned embodiment of the present application.

[0021] In the positioning method of the terminal device provided in the embodiment of the present application, according to the multimodal scene information corresponding to the terminal device, the target scene type in which the terminal device is located can be accurately determined. Then the target scene type is associated with the second positioning point corresponding to various candidate positioning methods to obtain at least one target second positioning point matching the target scene type. Finally, according to the distance and direction between the first positioning point and each target second positioning point, the matching positioning point with the highest matching degree with the first positioning point is determined in each target second positioning point, and the first positioning point is corrected using the matching positioning point. In this way, the embodiment of the present application obtains multiple second positioning points by adopting multiple candidate positioning methods. In the case where the positioning accuracy of the first positioning point of the target positioning method is low, the matching positioning point with the highest matching degree with the first positioning point in each second positioning point can be selected according to the target scene type in which the terminal device is located and the distance and direction between the first positioning point and each second positioning point to correct the first positioning point, thereby improving the accuracy of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 It is a flowchart of a method for positioning a terminal device provided by an embodiment of the present application;

[0024] Figure 2 It is a structural diagram of scene type identification provided by an embodiment of the present application;

[0025] Figure 3 is a structural schematic diagram of a positioning device for a terminal device provided in another embodiment of the present application;

[0026] Figure 4 It is a structural diagram of a positioning device of a terminal device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0027] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0028] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0029] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

[0030] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0031] In the related art, terminal devices are usually positioned based on a single positioning method such as WIFI positioning to provide reliable navigation. However, in complex scenarios or when the positioning algorithm fails, the positioning accuracy of a single positioning method such as WIFI positioning is low.

[0032] The purpose of the present application is to provide a positioning method, device, equipment, storage medium and program product for a terminal device. In the positioning method for a terminal device provided in an embodiment of the present application, according to the multimodal scene information corresponding to the terminal device, the target scene type in which the terminal device is located can be accurately determined. Then the target scene type is associated with the second positioning point corresponding to various candidate positioning methods to obtain at least one target second positioning point matching the target scene type. Finally, according to the distance and direction between the first positioning point and each target second positioning point, the matching positioning point with the highest matching degree with the first positioning point is determined in each target second positioning point, and the first positioning point is corrected using the matching positioning point. In this way, the embodiment of the present application obtains multiple second positioning points by adopting multiple candidate positioning methods. In the case where the positioning accuracy of the first positioning point of the target positioning method is low, the matching positioning point with the highest matching degree with the first positioning point in each second positioning point can be selected according to the target scene type in which the terminal device is located and the distance and direction between the first positioning point and each second positioning point to correct the first positioning point, thereby improving the accuracy of positioning.

[0033] The following describes specific embodiments of the terminal device positioning method, apparatus, device, storage medium and program product provided in the embodiments of the present application. First, the terminal device positioning method is introduced.

[0034] Figure 1 A flow chart of a method for positioning a terminal device is provided. The method for positioning a terminal device can be applied to a server. The method for positioning a terminal device can include the following S101 to S105.

[0035] S101, obtaining a first positioning point of a terminal device, multiple second positioning points, and multimodal scene information of a location of the terminal device, wherein the first positioning point is a positioning point obtained by a target positioning method, and the second positioning point is a positioning point obtained by a candidate positioning method other than the target positioning method.

[0036] In this embodiment, the first positioning point is the position obtained by the terminal device through the target positioning method. For example, in the navigation process, high-precision WIFI positioning technology is generally used for positioning. Therefore, in the navigation process, high-precision WIFI positioning technology is used as the target positioning method.

[0037] The second positioning point is a position obtained by positioning the terminal device through a candidate positioning method. For example, the candidate positioning method may be to estimate the positioning point by combining navigation, inertia or GPS direction features.

[0038] The multimodal scene information is information that characterizes the scene environment where the terminal device is located. For example, the multimodal scene information may include scene image information, scene audio information, scene temperature information, scene brightness information, and the like.

[0039] As an example, the server is in a complex scene, which causes the target positioning method to fail. In order to ensure the accuracy of the terminal device positioning, it is necessary to correct the first positioning point obtained by the target positioning method. At this time, the server obtains the first positioning point corresponding to the terminal device target positioning method, multiple second positioning points corresponding to the candidate positioning method, and multimodal scene information of the terminal device's location.

[0040] S102: Determine a target scene type of the terminal device according to the multimodal scene information corresponding to the terminal device.

[0041] In this embodiment, the scene types may include a bustling city center, a commercial area, a residential area, a traffic congested area, a quiet alley, and the like.

[0042] As an example, the server obtains the scene image information and scene audio information of the terminal device. The scene image information shows that the terminal device is in a remote path at this time, and the scene audio information shows that the ambient noise level is low at this time. Therefore, by combining the scene image information and the scene audio information, it can be determined that the terminal device is in a quiet alley at this time.

[0043] S103: Determine, according to the target scene type, from the second positioning points, a target second positioning point that matches the target scene type.

[0044] In this embodiment, according to the position of each second positioning point, the scene type corresponding to each second positioning point can be determined. Then the server matches the scene type corresponding to each second positioning point with the target scene type corresponding to the terminal device, and at least one target second positioning point in the same scene type as the terminal device can be obtained.

[0045] For example, the target scene type corresponding to the terminal device is a quiet alley, and five second positioning points A, B, C, D, and E can be obtained through the candidate positioning method. According to the position of each second positioning point, it is determined that A is located in the scene type of the commercial district, B is located in the scene type of the quiet alley, C is located in the scene type of the commercial district, D is located in the scene type of the quiet alley, and E is located in the scene type of the quiet alley. Therefore, it can be determined that B, D, and E are in the same scene type as the terminal device, that is, B, D, and E are the corresponding target second positioning points.

[0046] S104: Determine, according to the distance and direction between the first positioning point and the second positioning point of each target, a matching positioning point with the highest matching degree with the first positioning point among the second positioning points of each target.

[0047] In this embodiment, after obtaining the target second positioning point, the server can further select the target second positioning point with the same direction as the first positioning point and the closest distance as the matching positioning point according to the distance and direction between the first positioning point and each target second positioning point.

[0048] S105, correcting the first positioning point by matching the positioning points to obtain positioning information of the terminal device.

[0049] In this embodiment, as an example, the server may directly replace the current first positioning point with the matching positioning point, and use the positioning information of the matching positioning point as the current positioning information of the terminal device.

[0050] As another example, the server may also combine the first positioning point and the matching positioning point, and use the position of the midpoint of the line segment corresponding to the matching positioning point and the first positioning point as the corrected positioning information of the terminal device.

[0051] Through this embodiment, according to the multimodal scene information corresponding to the terminal device, the target scene type in which the terminal device is located can be accurately determined. Then the target scene type is associated with the second positioning point corresponding to various candidate positioning methods, and at least one target second positioning point matching the target scene type is obtained. Finally, according to the distance and direction between the first positioning point and each target second positioning point, the matching positioning point with the highest matching degree with the first positioning point is determined in each target second positioning point, and the first positioning point is corrected using the matching positioning point. In this way, the embodiment of the present application obtains multiple second positioning points by adopting multiple candidate positioning methods. In the case where the positioning accuracy of the first positioning point of the target positioning method is low, the matching positioning point with the highest matching degree with the first positioning point in each second positioning point can be selected according to the target scene type in which the terminal device is located and the distance and direction between the first positioning point and each second positioning point to correct the first positioning point, thereby improving the accuracy of positioning.

[0052] As an optional embodiment, the multimodal scene information includes scene image information and scene audio information;

[0053] S102 may specifically include:

[0054] Extracting features of scene image information to obtain image feature vectors corresponding to the scene image information;

[0055] Extracting features of the scene audio information to obtain an audio feature vector corresponding to the scene audio information;

[0056] The image feature vector and the audio feature vector are concatenated to obtain a scene feature vector corresponding to the multimodal scene information;

[0057] Input the scene feature vector into the scene recognition model to obtain the confidence level of each scene type corresponding to the terminal device;

[0058] The scene type with the highest confidence is determined as the target scene type in which the terminal device is located.

[0059] In this embodiment, the confidence level is used to characterize the probability of the corresponding predicted scene type output by the scene recognition model.

[0060] Feature extraction methods for scene image information can include image edge detection, image binarization, support vector machines, convolutional neural networks and many other methods.

[0061] The feature extraction methods of scene audio information may include time domain feature extraction, frequency domain feature extraction, time-frequency domain feature extraction, and statistical feature extraction.

[0062] Vector concatenation methods include the "+" operator, the extend() method, and concatenate() and other concatenation functions.

[0063] As an example, the server extracts the image feature vector corresponding to the scene image information through a convolutional neural network, then performs frequency domain feature extraction on the scene audio information to obtain an audio feature vector, and concatenates the image feature vector and the audio feature vector into a scene feature vector through a concatenation function.

[0064] Finally, the scene feature vector is input into a pre-trained scene recognition model to obtain the confidence levels corresponding to various scene types, and the scene type with the highest confidence level is determined as the target scene type where the terminal device is actually located.

[0065] Through this embodiment, feature extraction is performed on the scene image information and the scene audio information respectively, so that the scene feature vector finally obtained includes the image feature vector corresponding to the scene image information and the audio feature vector corresponding to the scene audio information. Finally, the audio feature vector is input into the pre-trained scene recognition model to determine the target scene type of the terminal device. In this way, the embodiment of the present application can achieve more comprehensive and accurate environmental perception through the acquisition of multimodal scene information, thereby improving the accuracy of scene type recognition.

[0066] As an optional embodiment, performing feature extraction on the scene image information to obtain an image feature vector corresponding to the scene image information may specifically include:

[0067] The scene image information is input into the feature extraction network of the image recognition model to obtain the image feature vector corresponding to the scene image information.

[0068] In this embodiment, the convolutional neural network has powerful feature extraction capabilities, so the convolutional neural network can be used to build an image recognition model.

[0069] As an example, the server first preprocesses the scene image information. Specifically, it can include adjusting the size to 648*648, brightness equalization processing and denoising. Then the preprocessed scene image information is convolved. The size of the convolution layer is 3*3, the interval is 1, and the activation function is ReLU. After a certain convolution, activation, and pooling, the image feature vector corresponding to the scene image information is output.

[0070] Through this embodiment, an image recognition model is constructed, and the feature extraction network of the image recognition model is used to extract the features of scene image information. In this way, since the image recognition model has a strong feature learning ability, the accuracy of scene image information feature extraction can be improved, and the accuracy of subsequent scene type recognition can be improved.

[0071] As an optional embodiment, the scene audio information is scene audio information within a preset time period;

[0072] Performing feature extraction on the scene audio information to obtain an audio feature vector corresponding to the scene audio information may specifically include:

[0073] Divide the scene audio information into time windows to determine the sound frequency characteristics corresponding to different moments;

[0074] The sound frequency features corresponding to different moments are combined in chronological order to obtain the audio feature vector corresponding to the scene audio information.

[0075] In this embodiment, the audio feature vector in the scene audio information is extracted by frequency domain feature extraction. Specifically, frequency domain feature extraction may include at least one of Fourier transform, short-time Fourier transform and Mel-frequency cepstrum coefficient.

[0076] As an example, the server divides the scene audio information within a preset period into time windows through short-time Fourier transform to obtain a number of short-time audio frames. Then, each short-time audio frame is subjected to Fourier transform to reveal the frequency components of the audio, thereby obtaining the sound frequency features corresponding to different moments, and converting the sound frequency features into vector values.

[0077] Finally, the sound frequency features are combined into an audio vector feature matrix in time order to obtain the audio feature vector corresponding to the scene audio information.

[0078] Through this embodiment, the scene audio information is divided into time windows, the sound frequency characteristics corresponding to different moments are determined, and the frequency domain feature extraction of the scene audio information is realized. In this way, by extracting the frequency domain features of the scene audio information, the essence of the signal can be revealed and the expression ability of the features can be enhanced, thereby improving the accuracy of the scene audio information feature extraction, and further improving the accuracy of the subsequent scene type recognition.

[0079] As an optional embodiment, after extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information, the positioning method of the terminal device may further include:

[0080] Input the audio feature vector into a feature enhancement network of an audio recognition model to obtain an enhanced feature vector;

[0081] The image feature vector and the audio feature vector are concatenated to obtain a scene feature vector corresponding to the multimodal scene information, which may specifically include:

[0082] The image feature vector and the enhanced feature vector are concatenated to obtain the scene feature vector corresponding to the multimodal scene information.

[0083] In this embodiment, in order to improve the association between contexts in the audio feature vector, the audio feature vector may also be processed by a recurrent neural network. For example, the recurrent neural network may include any one of a long short-term memory network and a gated recurrent unit.

[0084] As an example, after obtaining the audio feature vector, the server further processes the audio feature vector through the gated recurrent unit in the pre-trained audio recognition model to obtain the processed enhanced feature vector. Finally, the image feature vector and the enhanced feature vector are concatenated into a scene feature vector.

[0085] As another example, Figure 2 As shown, a structural schematic diagram of scene type recognition is provided. The multimodal scene information obtained by the server includes scene image information and scene audio information. First, the image feature vector is obtained by extracting features from the scene image information, and then the audio feature vector is obtained by extracting features from the scene audio information, and the audio feature vector is enhanced to obtain an enhanced feature vector. Then, the image feature vector and the enhanced feature vector are concatenated to obtain a scene feature vector, and finally, scene recognition is performed based on the scene feature vector to determine the corresponding target scene type.

[0086] Through this embodiment, after obtaining the audio feature vector corresponding to the scene audio information, the audio feature vector is further enhanced, so that the temporal dynamics and contextual basis of the scene audio information can be better understood, which can lay a solid foundation for subsequent scene type recognition, thereby improving the accuracy of scene type recognition.

[0087] As an optional embodiment, after extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information, the positioning method of the terminal device may further include:

[0088] The audio feature vector is input into the classification network of the audio recognition model to obtain the audio classification result corresponding to the audio feature vector.

[0089] In this embodiment, the classification network of the audio recognition model is used to identify the specific type of audio. For example, the specific type of audio can be one of a high ambient noise level, a medium ambient noise level, and a low ambient noise level.

[0090] As an example, after obtaining the audio feature vector, the server further inputs the audio feature vector into the classification network of the audio recognition model to obtain the specific type of the audio. Then, after obtaining the target scene type of the terminal device, the target scene type can be verified to be accurate based on the specific type of the audio.

[0091] Through this embodiment, after obtaining the audio feature vector corresponding to the scene audio information, the specific type of the audio is further identified, so that it is possible to verify whether the target scene type is accurate according to the specific type of the audio, thereby improving the accuracy of scene type identification.

[0092] As an optional embodiment, before inputting the audio feature vector into the classification network of the audio recognition model to obtain the audio classification result corresponding to the audio feature vector, the positioning method of the terminal device may further specifically include:

[0093] Acquire at least one audio training sample, where the audio training sample includes historical audio information and an audio classification label of the historical audio information;

[0094] For each audio training sample, perform the following steps:

[0095] Input the historical audio information into the target model to obtain the type prediction probability corresponding to the historical audio;

[0096] Determine the loss function value of the target model based on the type prediction probability and the audio classification label;

[0097] When the loss function value does not meet the training stop condition, the model parameters of the target model are adjusted to obtain an updated target model, and the historical audio information is returned to the target model to obtain the type prediction probability corresponding to the historical audio information until the training stop condition is met to obtain the audio recognition model.

[0098] In this embodiment, the audio training samples include historical audio information obtained by the terminal device within a historical time period and an audio classification label of the audio type to which the historical audio information belongs.

[0099] The target model is a pre-set model to be trained for building an audio recognition model. For example, the target model can be a model built by a convolutional neural network.

[0100] As an example, the server inputs historical audio information in the audio training sample into the target model to obtain the type prediction probabilities of various audio types corresponding to the historical audio information.

[0101] Then, the audio type to which the historical audio information actually belongs is determined based on the audio classification label of the historical audio information. The function loss value corresponding to the historical audio information can be determined based on the type prediction probabilities of various audio types corresponding to the historical audio information and the audio type to which the historical audio information actually belongs.

[0102] When the loss function value is greater than the preset loss threshold, the model parameters of the target model are adjusted according to the loss function value to obtain an updated target model. The historical audio information is input into the updated target model, and the target model is trained in a loop until the loss function value is less than or equal to the preset loss threshold, and the training is stopped to obtain an audio recognition model.

[0103] Through this embodiment, the target model is trained using historical audio information and audio classification labels of the historical audio information, thereby obtaining an audio recognition model. This helps to obtain the specific type of audio through the audio recognition model, thereby being able to verify whether the target scene type is accurate based on the specific type of audio, thereby improving the accuracy of scene type recognition.

[0104] As an optional embodiment, determining the loss function value of the target model according to the type prediction probability and the audio classification label may specifically include:

[0105] Get the target one-hot encoding corresponding to the audio classification label;

[0106] Determine the target calculation value corresponding to each audio type according to the type prediction probability corresponding to each audio type, the target one-hot encoding, and the type weight of the corresponding audio type;

[0107] According to the calculated values ​​of each target, the loss function value of the target model is obtained.

[0108] In this embodiment, one-hot encoding is a method for processing categorical data, which is mainly used to convert categorical data into numerical data. In one-hot encoding, each category value is converted into a binary vector whose length is equal to the number of categories, in which only one position has a value of 1, indicating the current category, and the values ​​of the remaining positions are all 0.

[0109] For example, suppose there are three types of audio: high ambient noise level, medium ambient noise level, and low ambient noise level. Given an audio classification label of "medium ambient noise level", its corresponding target unique hot encoding is [0, 1, 0].

[0110] Specifically, the loss function value of the target model can be determined by the following formula 1:

[0111]

[0112] Where l represents the loss function value of the target model. i represents the type weight of the i-th audio type, where w1+w2+w3=1. t represents the target one-hot encoding corresponding to the audio classification label, y i represents the type prediction probability of the i-th audio type.

[0113] Through this embodiment, the loss function value of the target model can be accurately calculated according to the type prediction probability corresponding to the audio type, the target one-hot encoding corresponding to the audio classification label, and the type weight of the audio type. This can help train the target model using the loss function value, thereby obtaining an accurate audio recognition model and improving the recognition accuracy of the audio type.

[0114] As an optional embodiment, before acquiring the multimodal scene information of the location of the terminal device, the positioning method of the terminal device may further include:

[0115] Calculating the position deviation between the first positioning point and each second positioning point respectively;

[0116] When at least one position deviation among the position deviations is greater than or equal to the deviation threshold, determining that the terminal device triggers a positioning correction mode;

[0117] S102 may specifically include:

[0118] When the terminal device triggers the positioning correction mode, the target scene type of the terminal device is determined according to the multimodal scene information of the terminal device.

[0119] In this embodiment, in response to the user inputting the navigation destination, the navigation process is entered, and the navigation path planning is performed through navigation. When the positioning of the terminal device is correct, the normal navigation function can be performed to provide path planning for pedestrians; when the positioning of the terminal device is wrong, the positioning correction mode is triggered to determine the target scene type of the terminal device, so as to correct the first positioning point and obtain the positioning information of the terminal device.

[0120] Specifically, after obtaining the first positioning point and each second positioning point, the server will calculate the position deviation between the first positioning point and each second positioning point. When the position deviation between the first positioning point and the second positioning point is greater than or equal to the deviation threshold, it is confirmed that the positioning of the terminal device is wrong, thereby triggering the positioning correction mode. At this time, the terminal device will obtain the corresponding multimodal scene information to determine the target scene type in which the terminal device is located.

[0121] The position deviation between the first positioning point and the second positioning point can be determined by the following formula 2:

[0122]

[0123] Where D represents the position deviation between the first positioning point and the second positioning point, x actual ,y actual are the horizontal and vertical coordinates of the first positioning point, respectively, estimated ,y estimated are the horizontal and vertical coordinates of the second positioning point respectively.

[0124] Through this embodiment, according to the position deviation between the first positioning point and each second positioning point, it is determined whether the terminal device needs to trigger the positioning correction mode. In this way, only when the positioning correction mode needs to be triggered, the multimodal scene information corresponding to the terminal device is obtained to correct the first positioning point, thereby improving the efficiency of navigation positioning.

[0125] As an optional embodiment, S104 may specifically include:

[0126] According to the direction between the first positioning point and the second positioning point of each target, determine a candidate positioning point having the same direction as the first positioning point from the second positioning points of each target;

[0127] According to the distance between the first positioning point and each candidate positioning point, a matching positioning point having the shortest distance to the first positioning point is determined from each candidate positioning point.

[0128] In this embodiment, after obtaining the target second positioning point, the server further screens the target second positioning point according to the direction between the first positioning point and each target second positioning point, thereby obtaining candidate positioning points with the same direction as the first positioning point.

[0129] Finally, the distance between each candidate positioning point and the first positioning point is calculated, and the candidate positioning point with the shortest distance is selected as the matching positioning point.

[0130] Through this embodiment, according to the distance and direction between the first positioning point and each target second positioning point, the second positioning point with the most matching direction and distance is selected as the matching positioning point among each target second positioning point, and the first positioning point is corrected using the matching positioning point, thereby improving the accuracy of positioning.

[0131] Based on the positioning method of the terminal device. Accordingly, the present application also provides a specific embodiment of the positioning device of the terminal device.

[0132] like Figure 3 As shown, the positioning device 300 of the terminal device provided in the embodiment of the present application includes an information acquisition module 310, a type determination module 320, a positioning point matching module 330 and a positioning correction module 340.

[0133] The information acquisition module 310 is used to obtain a first positioning point of a terminal device, multiple second positioning points and multimodal scene information of the terminal device's location, wherein the first positioning point is a positioning point obtained by a target positioning method, and the second positioning point is a positioning point obtained by a candidate positioning method other than the target positioning method.

[0134] The type determination module 320 is used to determine the target scene type of the terminal device according to the multimodal scene information corresponding to the terminal device.

[0135] The positioning point matching module 330 is used to determine, according to the target scene type, a target second positioning point that matches the target scene type from among the second positioning points.

[0136] The positioning point matching module 330 is further used to determine the matching positioning point with the highest matching degree with the first positioning point among the second positioning points of each target according to the distance and direction between the first positioning point and the second positioning point of each target.

[0137] The positioning correction module 340 is used to correct the first positioning point by matching the positioning points to obtain the positioning information of the terminal device.

[0138] As an optional embodiment, the multimodal scene information includes scene image information and scene audio information;

[0139] The type determination module 320 specifically includes the following units:

[0140] An image feature extraction unit, used to extract features from scene image information to obtain an image feature vector corresponding to the scene image information;

[0141] An audio feature extraction unit, used to extract features from scene audio information to obtain an audio feature vector corresponding to the scene audio information;

[0142] A vector concatenation unit, used to concatenate the image feature vector and the audio feature vector to obtain a scene feature vector corresponding to the multimodal scene information;

[0143] A confidence determination unit, used to input the scene feature vector into the scene recognition model to obtain the confidence of each scene type corresponding to the terminal device;

[0144] The type determination unit is used to determine the scene type with the highest confidence as the target scene type of the terminal device.

[0145] As an optional embodiment, the image feature extraction unit is specifically used for:

[0146] The scene image information is input into the feature extraction network of the image recognition model to obtain the image feature vector corresponding to the scene image information.

[0147] As an optional embodiment, the scene audio information is scene audio information within a preset time period;

[0148] The audio feature extraction unit specifically includes the following subunits:

[0149] The audio division subunit is used to divide the scene audio information into time windows and determine the sound frequency characteristics corresponding to different moments;

[0150] The feature combination subunit is used to combine the sound frequency features corresponding to different moments in time order to obtain the audio feature vector corresponding to the scene audio information.

[0151] As an optional embodiment, after extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information, the type determination module 320 specifically further includes the following units:

[0152] A feature enhancement unit, used for inputting the audio feature vector into a feature enhancement network of an audio recognition model to obtain an enhanced feature vector;

[0153] The vector concatenation unit is specifically used for:

[0154] The image feature vector and the enhanced feature vector are concatenated to obtain the scene feature vector corresponding to the multimodal scene information.

[0155] As an optional embodiment, after extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information, the positioning device 300 of the terminal device specifically further includes the following modules:

[0156] The audio classification module is used to input the audio feature vector into the classification network of the audio recognition model to obtain the audio classification result corresponding to the audio feature vector.

[0157] As an optional embodiment, before inputting the audio feature vector into the classification network of the audio recognition model to obtain the audio classification result corresponding to the audio feature vector, the positioning device 300 of the terminal device specifically further includes the following modules:

[0158] A sample acquisition module, used to acquire at least one audio training sample, where the audio training sample includes historical audio information and audio classification labels of the historical audio information;

[0159] The sample training module is used to execute the following units for each audio training sample:

[0160] A probability determination unit, used to input historical audio information into a target model to obtain a type prediction probability corresponding to the historical audio;

[0161] A loss value determination unit, used to determine a loss function value of a target model according to a type prediction probability and an audio classification label;

[0162] The parameter adjustment unit is used to adjust the model parameters of the target model when the loss function value does not meet the training stop condition to obtain an updated target model, and return the historical audio information to the target model to obtain the type prediction probability corresponding to the historical audio information until the training stop condition is met to obtain the audio recognition model.

[0163] As an optional embodiment, the loss value determination unit specifically includes the following subunits:

[0164] The encoding acquisition subunit is used to obtain the target one-hot encoding corresponding to the audio classification label;

[0165] A calculation value determination subunit, used to determine the target calculation value corresponding to each audio type according to the type prediction probability corresponding to each audio type, the target one-hot encoding and the type weight of the corresponding audio type;

[0166] The loss value determination subunit is used to obtain the loss function value of the target model according to the target calculation values.

[0167] As an optional embodiment, before acquiring the multimodal scene information of the location of the terminal device, the positioning device 300 of the terminal device specifically further includes the following modules:

[0168] A deviation calculation module, used to calculate the position deviation between the first positioning point and each second positioning point respectively;

[0169] A correction trigger module, used to determine that the terminal device triggers a positioning correction mode when at least one position deviation among the position deviations is greater than or equal to a deviation threshold;

[0170] The type determination module 320 is specifically used for:

[0171] When the terminal device triggers the positioning correction mode, the target scene type of the terminal device is determined according to the multimodal scene information of the terminal device.

[0172] As an optional embodiment, the positioning point matching module 330 specifically includes the following units:

[0173] A first matching unit is used to determine, from the second positioning points of each target, a candidate positioning point with the same direction as the first positioning point according to the direction between the first positioning point and the second positioning point of each target;

[0174] The second matching unit is used to determine, from among the candidate positioning points, a matching positioning point with the shortest distance to the first positioning point according to the distance between the first positioning point and the candidate positioning points.

[0175] Based on the positioning method of the terminal device. Accordingly, the present application also provides a specific embodiment of the positioning device of the terminal device.

[0176] Figure 4 A schematic diagram of the hardware structure of a positioning device of a terminal device provided in an embodiment of the present application is shown.

[0177] The positioning device of the terminal device may include a processor 401 and a memory 402 storing computer program instructions.

[0178] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0179] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0180] The processor 401 implements any one of the terminal device positioning methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0181] In one example, the positioning device of the terminal device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0182] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0183] Bus 410 includes hardware, software or both, and the parts of the positioning device of terminal equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0184] In addition, in combination with the terminal device positioning method in the above embodiments, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any terminal device positioning method in the above embodiments is implemented.

[0185] In addition, in combination with the terminal device positioning method in the above embodiment, the present application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device executes any terminal device positioning method in the above embodiment.

[0186] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0187] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0188] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0189] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0190] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for positioning a terminal device, characterized in that: The method comprises: Acquire a first positioning point of a terminal device, multiple second positioning points, and multimodal scene information of a location of the terminal device, wherein the first positioning point is a positioning point obtained by a target positioning method, and the second positioning point is a positioning point obtained by a candidate positioning method other than the target positioning method; Determining the target scene type of the terminal device according to the multimodal scene information corresponding to the terminal device; According to the target scene type, determining a target second positioning point matching the target scene type from each of the second positioning points; Determine, according to the distance and direction between the first positioning point and each of the target second positioning points, a matching positioning point with the highest matching degree with the first positioning point among each of the target second positioning points; The first positioning point is corrected through the matching positioning point to obtain the positioning information of the terminal device.

2. The method according to claim 1, characterized in that The multimodal scene information includes scene image information and scene audio information; The determining, according to the multimodal scene information corresponding to the terminal device, the target scene type in which the terminal device is located includes: Extracting features from the scene image information to obtain an image feature vector corresponding to the scene image information; Extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information; Concatenating the image feature vector with the audio feature vector to obtain a scene feature vector corresponding to the multimodal scene information; Inputting the scene feature vector into a scene recognition model to obtain the confidence level of each scene type corresponding to the terminal device; The scene type with the highest confidence is determined as the target scene type in which the terminal device is located.

3. The method according to claim 2, characterized in that The extracting features of the scene image information to obtain an image feature vector corresponding to the scene image information includes: The scene image information is input into a feature extraction network of an image recognition model to obtain the image feature vector corresponding to the scene image information.

4. The method according to claim 2, characterized in that: The scene audio information is scene audio information within a preset time period; The extracting features of the scene audio information to obtain an audio feature vector corresponding to the scene audio information includes: Dividing the scene audio information into time windows to determine the sound frequency characteristics corresponding to different moments; The sound frequency features corresponding to the different moments are combined in time order to obtain the audio feature vector corresponding to the scene audio information.

5. The method according to claim 2, characterized in that: After extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information, the method further includes: Inputting the audio feature vector into a feature enhancement network of an audio recognition model to obtain an enhanced feature vector; The step of concatenating the image feature vector and the audio feature vector to obtain a scene feature vector corresponding to the multimodal scene information includes: The image feature vector and the enhanced feature vector are concatenated to obtain a scene feature vector corresponding to the multimodal scene information.

6. The method according to claim 2, characterized in that After extracting features from the scene audio information to obtain an audio feature vector corresponding to the scene audio information, the method further includes: The audio feature vector is input into a classification network of an audio recognition model to obtain an audio classification result corresponding to the audio feature vector.

7. The method according to claim 6, characterized in that Before inputting the audio feature vector into a classification network of an audio recognition model to obtain an audio classification result corresponding to the audio feature vector, the method further includes: Acquire at least one audio training sample, where the audio training sample includes historical audio information and an audio classification label of the historical audio information; For each of the audio training samples, the following steps are performed respectively: Inputting the historical audio information into the target model to obtain the type prediction probability corresponding to the historical audio; Determining a loss function value of the target model according to the type prediction probability and the audio classification label; When the loss function value does not meet the training stop condition, the model parameters of the target model are adjusted to obtain the updated target model, and the historical audio information is returned to the target model to obtain the type prediction probability corresponding to the historical audio information until the training stop condition is met to obtain the audio recognition model.

8. The method according to claim 7, characterized in that The determining the loss function value of the target model according to the type prediction probability and the audio classification label includes: Obtain a target one-hot encoding corresponding to the audio classification label; Determine target calculation values ​​corresponding to various audio types according to the type prediction probabilities corresponding to various audio types, the target one-hot encodings, and the type weights of the corresponding audio types; According to each of the target calculation values, the loss function value of the target model is obtained.

9. The method according to any one of claims 1 to 8, characterized in that: Before acquiring the multimodal scene information of the location of the terminal device, the method further includes: respectively calculating the position deviations between the first positioning point and each of the second positioning points; In the case where at least one of the position deviations is greater than or equal to the deviation threshold, determining that the terminal device triggers a positioning correction mode; The determining, according to the multimodal scene information of the terminal device, the target scene type in which the terminal device is located includes: When the terminal device triggers the positioning correction mode, the target scene type of the terminal device is determined according to the multimodal scene information of the terminal device.

10. The method according to any one of claims 1 to 8, characterized in that: The determining, according to the distance and direction between the first positioning point and each of the target second positioning points, a matching positioning point with the highest matching degree with the first positioning point among each of the target second positioning points comprises: According to the direction between the first positioning point and each of the target second positioning points, determine a candidate positioning point having the same direction as the first positioning point from each of the target second positioning points; According to the distance between the first positioning point and each of the candidate positioning points, a matching positioning point having the shortest distance to the first positioning point is determined from each of the candidate positioning points.

Images