Equipment positioning method and device, electronic equipment and storage medium

Abstract

The invention provides an equipment positioning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining query information related to a range where target equipment is located; according to a matching result of the query information and a pre-constructed feature semantic map, first range information of the target device is determined, and the first range information represents a range where the target device is located; target sensor data of the target equipment is processed according to the map data in the first range information to obtain second position information of the target equipment, the second position information represents the position of the target equipment in the range pointed by the first range information, and the target sensor data comprises at least one of laser radar data and image data.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a method, apparatus, electronic device and storage medium for device positioning. Background Art

[0002] With the development of electronic information technology, a variety of devices that can move autonomously in a specific environment (indoor or outdoor) to perform specific tasks have appeared in daily life, such as sweeping robots, food delivery robots, etc. When these devices move in a specific environment, sometimes they lose their positioning. When the positioning is lost, the device cannot determine its own position in the environment, resulting in the device being unable to plan subsequent movement routes based on its own position.

[0003] In related technologies, when positioning is lost, the device can use its own camera to capture images of the surrounding environment, match the captured images with a pre-established map of the environment, and re-determine its own position based on the matching results. This method is only based on image matching, but the image perspective used to build the map is single, and this method uses only image data for matching, resulting in a low matching success rate, and sometimes the location of the device cannot be determined. Summary of the invention

[0004] In view of this, the present application provides a method, apparatus, electronic device and storage medium for device positioning to solve the problem of low positioning success rate in the prior art.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] A first aspect of the present application provides a method for positioning a device, comprising:

[0007] Obtain query information related to the range of the target device;

[0008] Determine first range information of the target device according to a matching result between the query information and a pre-built feature semantic map, where the first range information represents a range where the target device is located;

[0009] The target sensor data of the target device is processed according to the map data within the first range information to obtain the second position information of the target device, wherein the second position information represents the position of the target device within the range indicated by the first range information, and the target sensor data includes at least one of lidar data and image data.

[0010] Optionally, the feature semantic map includes multiple coordinate data and visual features corresponding to the coordinate data, the coordinate data represents the position of an object in the environment where the target device is located, and the visual features corresponding to the coordinate data are visual features of the corresponding object.

[0011] Optionally, the method of constructing the feature semantic map includes:

[0012] Obtaining multiple coordinate data of the environment in which the target device is located, and multiple frames of key images obtained by photographing the environment;

[0013] Processing the key image according to the multimodal large model to obtain pixel features corresponding to each pixel in the key image;

[0014] For each of the coordinate data, according to the mapping relationship between the pixel points contained in the key image and the coordinate data, the pixel features of each pixel point corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

[0015] Optionally, the step of processing the key image according to the multimodal large model to obtain a pixel feature corresponding to each pixel in the key image includes:

[0016] Extracting global image features and local image features of the key image according to the multimodal large model;

[0017] The global image feature and the local image feature of the local area where the pixel point in the key image is located are fused to obtain the pixel feature corresponding to the pixel point in the key image.

[0018] Optionally, fusing pixel features of each pixel point corresponding to the coordinate data to obtain visual features corresponding to the coordinate data includes:

[0019] Filter out a key area pixel feature whose corresponding pixel point is located in a key area of ​​the image from the plurality of pixel features;

[0020] Filtering out pixel features related to the occluded object from the pixel features of the key area to obtain filtered pixel features;

[0021] The filtered pixel features of the pixel points corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

[0022] Optionally, the query information includes at least one of image information and text information, the image information is obtained by photographing an object within a range of the target device, and the text information is obtained by a human-computer interaction module of the target device.

[0023] Optionally, determining the first range information of the target device according to the matching result of the query information and the pre-built feature semantic map includes:

[0024] In the case where the query information includes the image information, extracting global image features and key object features of the image information according to the multimodal macro model;

[0025] fusing the global image features and the key object features to obtain query image features;

[0026] The first range information of the target device is determined according to the matching result between the query image feature and the feature semantic map.

[0027] A second aspect of the present application provides a device for positioning a device, including:

[0028] An obtaining unit, used to obtain query information related to the range where the target device is located;

[0029] A determination unit, configured to determine first range information of the target device according to a matching result of the query information and a pre-built feature semantic map, wherein the first range information represents a range where the target device is located;

[0030] A processing unit is used to process the target sensor data of the target device according to the map data within the first range information to obtain the second position information of the target device, wherein the second position information represents the position of the target device within the range indicated by the first range information, and the target sensor data includes at least one of lidar data and image data.

[0031] A third aspect of the present application provides an electronic device, including:

[0032] one or more processors;

[0033] a storage device having one or more programs stored thereon;

[0034] When the one or more programs are executed by the one or more processors, the one or more processors implement the device positioning method provided in any one of the first aspects of the present application.

[0035] The electronic device in the third aspect can be a target device. The target device can obtain query information and report the query information to a server used to run a multimodal large model. The server obtains first range information based on the query information and sends the first range information to the target device. After the target device obtains the first range information, it determines the second location information based on the map data and target sensor data in the first range information.

[0036] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for device positioning provided in any one of the first aspects of the present application is implemented.

[0037] The beneficial effects of this solution are:

[0038] First, the first range information of the target device is determined based on the query information and feature semantic map related to the range where the target device is located. Then, based on the first range information, the map data within the corresponding range, as well as the laser radar data and image data of the target device are used to accurately locate the target device, and the second position information of the target device within the corresponding range is obtained. In the above manner, this solution can combine multi-modal data, i.e., the laser radar data and image data of the target device, to locate the target device after the target device loses its position, thereby improving the success rate of positioning. Furthermore, the above query information may include image information and text information. In this case, this solution can combine multi-modal query information to further improve the success rate of re-positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A flowchart of a method for positioning a device disclosed in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of first range information and second location information disclosed in an embodiment of the present application;

[0042] Figure 3 A flowchart of a method for constructing a feature semantic map disclosed in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of a device for positioning a device disclosed in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In this application, the terms "comprises", "comprising" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0047] Furthermore, in this document, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0048] This application embodiment provides a method for locating a device. Figure 1 , is a flow chart of the method, and the method may include the following steps.

[0049] S101, obtaining query information related to the range where the target device is located.

[0050] S102: Determine first range information of the target device according to the query information and the matching result of the pre-built feature semantic map, where the first range information represents the range where the target device is located.

[0051] S103, processing the target sensor data of the target device according to the map data within the first range information to obtain the second position information of the target device, the second position information represents the position of the target device within the range indicated by the first range information, and the target sensor data includes at least one of the laser radar data and the image data.

[0052] The execution subject of the above-mentioned device positioning method can be a target device where positioning loss occurs. In this case, after determining that its positioning has been lost, the target device can execute step S101 to obtain query information, and then the target device executes S102 to send the query information to a server that is connected to the target device in communication. After receiving the query information, the server matches the query information with the feature semantic map, determines the first range information based on the matching result, and sends the first range information to the target device. After the target device obtains the first range information, it can directly execute S103 based on the first range information, that is, process the target sensor data locally on the target device to obtain the second location information, or, after obtaining the first range information, the target device sends the target sensor data to the server to trigger the server to process the target sensor data in the manner of step S103, and then receive the second location information fed back by the server.

[0053] The execution subject of the above-mentioned device positioning method can be a server. In this case, in step S101, the server can receive the query information reported by the target device, and the query information can be reported to the server after the target device determines that it has lost its positioning. After receiving the query information, the server executes step S102 based on the query information, matches the query information with the feature semantic map to obtain the first range information, and then the server executes step S103 based on the first range information, determines the second location information of the target device, and sends the second location information to the target device. Among them, the target sensor data can be sent to the server by the target device together with the query information in S101, or it can be sent to the server by the target device after the server determines the first range information.

[0054] The target device of this embodiment can be any device that can autonomously move along a specific route and perform corresponding tasks within a certain range of environments (for example, indoor environments such as living rooms and classrooms, and outdoor environments such as playgrounds and courts). For example, the target device can be a sweeping robot-type device.

[0055] The target device may determine whether it has lost its positioning by any existing method, which is not limited in this embodiment.

[0056] The first range information may be any information that can characterize the range of the target device. In some embodiments, the first range information may include the location information of an object in the environment as the center of a circle and the distance information as the radius. In this case, the range indicated by the first range information may be a circular range with the location of the object as the center and the distance information as the radius. Generally, the accuracy of the first range information may be within 1m.

[0057] The second location information may include any information that can accurately locate the location of the target device. For example, the second location information may include the orientation information and distance information of the target device relative to an object in the environment. Generally, the accuracy of the second location information can reach centimeter level or higher.

[0058] As an example, see Figure 2 , the first range information may include: a radius of 1 meter, the center of the circle is the location of object A (Xa, Ya), based on the first range information, it can be determined that the target device is currently within 1 meter around object A. The second location information may include: the target device is located in the east direction of object A, 0.3 meters away from object A, based on the second location information, it can be accurately located at the location of the target device Figure 2 At the position P shown.

[0059] The object A may be any object in the environment where the target device is located. For example, the object A may be a desk in a classroom, a roadblock placed on a playground, or a shelf in a supermarket.

[0060] The beneficial effects of this embodiment are:

[0061] First, the first range information of the target device is determined based on the query information and feature semantic map related to the range where the target device is located. Then, based on the first range information, the map data within the corresponding range, as well as the laser radar data and image data of the target device are used to accurately locate the target device, and the second position information of the target device within the corresponding range is obtained. In the above manner, this solution can combine multi-modal data, i.e., the laser radar data and image data of the target device, to locate the target device after the target device loses its position, thereby improving the success rate of positioning. Furthermore, the above query information may include image information and text information. In this case, this solution can combine multi-modal query information to further improve the success rate of re-positioning.

[0062] The method provided in this embodiment may be executed by a target device. After determining that positioning loss has occurred, the target device may execute step S101 to obtain query information (text or image), and then the target device sends the query information to the server to trigger the server to process the obtained query information using a multimodal large model to obtain first range information, and then the target device receives the first range information representing the approximate position fed back by the server (equivalent to step S102), and then the target device obtains more accurate second position information based on the first range information and target sensor data.

[0063] Before applying the method of this embodiment to locate the device, the server can Figure 3 The method shown builds a feature semantic map of the environment in which the target device is located.

[0064] The feature semantic map may include point cloud data of the environment in which the target device is located (hereinafter referred to as the environment) and visual features corresponding to the point cloud data. Specifically, the point cloud data may include multiple coordinate data, which are three-dimensional coordinate data, specifically the three-dimensional spatial coordinates of a point on the surface of any object in the environment. Therefore, these coordinate data can characterize the position of the object in the environment in which the target device is located. For example, the point cloud data may include multiple coordinate data, which correspond to multiple points on the surface of a sofa in a living room environment. The position of the sofa in the living room environment can be determined by combining the coordinate data of multiple points on the surface of the sofa.

[0065] Each coordinate data corresponds to a visual feature, and the visual feature can characterize the appearance of the corresponding object at that point, that is, the visual feature corresponding to the coordinate data is specifically the visual feature of the corresponding object. In combination with the above example, for multiple coordinate data corresponding to multiple points on the sofa surface, the visual feature of each coordinate data reflects the appearance of the corresponding point on the sofa surface.

[0066] See also Figure 3 , the ways to construct feature semantic maps can include:

[0067] A1, obtaining multiple coordinate data of the environment in which the target device is located, and multiple frames of key images obtained by shooting the environment;

[0068] A2, processing the key image according to the multimodal large model to obtain the pixel features corresponding to each pixel in the key image;

[0069] A3, for each coordinate data, according to the mapping relationship between the pixel points contained in the key image and the coordinate data, the pixel features of each pixel point corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

[0070] On the one hand, the server needs to obtain point cloud data of the environment, that is, multiple coordinate data of the environment. The server can obtain point cloud data of the environment through radar scanning or image reconstruction, or can use both methods to obtain point cloud data.

[0071] For the radar scanning solution, the server can be connected to a scanning device in the environment and equipped with a lidar, receive scanning data fed back by the scanning device, and determine the point cloud data of the environment based on the scanning data and the position and posture of the lidar itself.

[0072] For the image reconstruction solution, the server can receive multiple frames of images taken by a depth camera of a scanning device in the environment, as well as the position and posture of the depth camera when taking the images, and use any existing image reconstruction method, such as the Neural Radiance Field in 3D Vision (Nerf) method or the Simultaneous Localization and Mapping (SLAM) method to process the position and image of the depth camera to obtain point cloud data of the environment.

[0073] The above method for determining point cloud data can be referred to in the relevant prior art and will not be described in detail.

[0074] On the other hand, the server needs to obtain multiple frames of key images of the environment and the camera pose corresponding to each frame of the key image. The camera pose refers to the position and posture of the depth camera when shooting the corresponding frame of the image. If the server obtains point cloud data through a lidar, the server also needs to call the depth camera of the scanning device to shoot multiple frames of key images at a certain shooting interval, and then receive the key images and corresponding camera poses reported by the scanning device. If the server obtains point cloud data through image reconstruction, the server can select several frames of images as key images from the multiple frames provided by the scanning device according to a specific algorithm. The methods of shooting key images and selecting key images can be referred to the relevant existing technologies and will not be repeated here.

[0075] In step A2, after obtaining multiple frames of key images, the server can use a pre-built multimodal large model with image processing and feature extraction capabilities to process each frame of the key image to obtain the pixel features corresponding to each pixel point in each frame of the key image, and then obtain the visual features corresponding to the coordinate data based on the pixel features.

[0076] Optionally, in order to make the obtained visual features more accurately represent the appearance of the object at the point corresponding to the coordinate data, for each frame of the key image, the server can obtain the pixel features of the frame of the key image in the following manner:

[0077] Extracting global image features and local image features of key images based on a multimodal large model;

[0078] The global features of the image are fused with the local features of the image where the pixel points in the key image are located to obtain the pixel features corresponding to the pixel points in the key image.

[0079] After a frame of key image is processed by the multimodal large model, a global image feature and multiple local image features can be obtained. The global image feature can characterize the number, type, position and other information of the objects displayed in the whole frame of key image, while the local image features reflect the appearance attribute information of the objects displayed in different local areas in the frame of key image, such as the color, shape, material and other information of the objects.

[0080] For any pixel point in any frame of the key image, the multimodal large model can determine which local area the pixel point belongs to in the key image and where it is located in the local area. Then, the multimodal large model can fuse the position of the pixel point in the local area, the local image features of the local area where the pixel point is located, and the global image features of the key image of the frame to obtain the pixel features corresponding to the pixel point.

[0081] In step A3, the server also needs to determine the mapping relationship between the pixels in the key image and the coordinate data in the point cloud data. The mapping relationship between a certain pixel point and a certain coordinate data can be understood as the pixel point obtained by projecting the point corresponding to the coordinate data on the object surface onto the imaging plane of the camera. In other words, if a mapping relationship exists between a certain pixel point P1 and a certain coordinate data, it means that P1 is the projection of the point P2 on the object surface corresponding to the coordinate data onto the plane.

[0082] In step A3, the server can project each coordinate data in the point cloud data onto the pixel points of each key image frame according to the camera pose corresponding to each key image frame obtained in advance, thereby determining the mapping relationship between the coordinate data and the pixel points, that is, determining which pixel points are respectively projected by which coordinate data. The method for determining the above mapping relationship can refer to the relevant prior art and will not be repeated here.

[0083] After determining the mapping relationship between pixel points and coordinate data, for each coordinate data, the server can directly fuse the pixel features of multiple pixel points corresponding to the coordinate data, and use the result as the visual feature corresponding to the coordinate data. The fusion method can be to add multiple pixel features according to a certain weight.

[0084] Alternatively, in order to improve the accuracy of the obtained visual features, the server may also first post-process the pixel features in the following manner, and then fuse the post-processed pixel features of the pixel points corresponding to the coordinate data to obtain the visual features corresponding to the coordinate data:

[0085] B1, selecting the key area pixel features whose corresponding pixel points are located in the key area of ​​the image from multiple pixel features;

[0086] B2, filtering out pixel features related to the occluded object from the pixel features of the key area to obtain filtered pixel features;

[0087] B3, fusing the filtered pixel features of the pixel points corresponding to the coordinate data to obtain the visual features corresponding to the coordinate data.

[0088] The key image area in step B1 can be a rectangular or circular area within a certain distance from the center of the key image. As an example, assuming that the key image is an image with a resolution of 1080*720, the key image area can be a rectangular area of ​​540*360 located in the center of the image. The key area pixel features corresponding to the pixel points located in the key image area can be screened out by deleting the pixel features of the pixel points that are not located in the key image area, and only retaining the pixel features of the pixel points located in the key image area of ​​the image, such as the pixel features of the pixel points located in the center of the above-mentioned key image. The retained pixel features are called key area pixel features.

[0089] The purpose of filtering by B1 is that in the image captured by the depth camera, the pixels at the edge of the image may be severely distorted and cannot truly reflect the appearance of the corresponding object surface. Deleting the pixel features of these pixels can improve the accuracy of the visual features subsequently obtained by fusion of pixel features.

[0090] In step B2, the server may identify which objects in the key image are blocked by using relevant prior art techniques, and determine the blocked objects as blocked objects.

[0091] For example, object A can be identified in key image 1, but in another key image 2, object A is blocked by object B, so the server can determine that object A is a blocked object.

[0092] The pixel features related to the occluded object refer to the pixel features of the pixels belonging to the occluded object in the key image. In combination with the above example, after determining that object A is an occluded object, the pixel features of the pixels belonging to object A in key image 1 are the pixel features related to the occluded object A.

[0093] After identifying the occluded object, the server may delete the pixel features of the pixels of the occluded object from among all the retained pixel features of the key area, and use the remaining pixel features that have not been deleted as the filtered pixel features of step B2.

[0094] In combination with the above example, after determining that object A is an occluded object, the server may delete the pixel features of the pixel points belonging to object A in key image 1.

[0095] The purpose of filtering out pixel features related to occluded objects is to:

[0096] When the visual features reflect the appearance of an obstructed object in the distance, matching the query information with the visual features may result in a successful match with the distant object, which in turn may cause the identified first range information to be inaccurate. For example, the target device is actually located near object B, but the query information and the visual features of object A in the distance that is obstructed by object B are successfully matched. This may cause the server to identify that the target device is near object A. In this case, the obtained first range information is inaccurate. Deleting the pixel features related to the obstructed object can ensure that the visual features obtained by pixel feature fusion only reflect the appearance of unobstructed objects that are closer to the target device, thereby avoiding the above situation to a certain extent.

[0097] After post-processing is performed in the manner of steps B1 and B2 to obtain the filtered pixel features, the server can fuse the filtered pixel features of the pixel points corresponding to each coordinate data to obtain the visual features corresponding to the coordinate data.

[0098] The pixel point corresponding to the coordinate data refers to the pixel point that has a mapping relationship with the coordinate data. As an example, the server projects each coordinate data in the point cloud data to each key image frame according to the camera pose corresponding to each key image frame obtained in advance, and determines that there is a mapping relationship between coordinate data 1 and pixel point 1 of key image 1, pixel point 2 of key image 2, and pixel point 3 of key image 3, and after the above post-processing, the pixel features corresponding to pixels 1 to 3 are retained as filtered pixel features, so the server can fuse pixel feature 1 corresponding to pixel point 1, pixel feature 2 corresponding to pixel point 2, and pixel feature 3 corresponding to pixel point 3 to obtain visual feature 1 corresponding to coordinate data 1.

[0099] In this embodiment, the query information may include at least one of image information and text information. The image information is obtained by photographing an object within a range of the target device, and the text information is obtained by a human-computer interaction module of the target device.

[0100] When the query information includes image information, if the target device finds that its positioning has been lost and needs to obtain query information, it can stop in place, control its own depth camera to rotate one circle, capture multiple frames of images of the surroundings, and upload these images to the server as query information.

[0101] In the case where the query information includes text information, when the target device finds that it has lost its position and needs to obtain the query information, the target device can output prompt information through its own human-computer interaction module, and the prompt information is used to prompt the current user to input information that can characterize the range of the target device. Then, the human-computer interaction module collects the feedback provided by the user after the prompt information is output, and obtains the text information as the query information based on the user's feedback. Exemplarily, the user's feedback can be voice, and its content can be "Currently located near the second table on the left side of the first row", and the target device can perform voice recognition on the above voice, obtain the content therein as query information and report it to the server; or, the user's feedback can also include text directly entered by the user through the keyboard or touchpad, in which case the target device can directly report the text as query information to the server.

[0102] The human-computer interaction module may include a display screen, a speaker, a microphone, a button, a touch pad, and other modules that can realize human-computer interaction.

[0103] After reporting the query information to the server, if the query information includes image information, the server may match the query information with the feature semantic map in the following manner to obtain the first range information:

[0104] C1, when the query information includes image information, extract global image features and key object features of the image information based on the multimodal large model;

[0105] C2, the query image features are obtained by fusing the global image features and the key object features;

[0106] C3, determining first range information of the target device according to the matching result between the query image feature and the feature semantic map.

[0107] In step C1, on the one hand, the server can use the multimodal large model to extract the global image features of the image information. When the image information includes multiple frames of images, such as the multiple frames of images obtained by the target device rotating the depth camera for one circle in the above example, the server can extract the global image features of each frame of the image respectively, and then fuse the global image features of the multiple frames of images to obtain the global image features of the image information.

[0108] On the other hand, the server can use the multimodal large model to identify key objects displayed in the image information, and then extract the image features of these key objects as key object features. Which objects are identified as key objects can be determined by the multimodal large model. Generally, the multimodal large model can identify objects with larger sizes and closer to the target device as target objects.

[0109] In step C2, the server may perform weighted summation of the global image features of the image information and the key object features of the image information according to certain weights, and use the obtained result as the fused query image feature.

[0110] In step C3, the server may respectively calculate the cosine similarity between the query image feature and the visual feature corresponding to each coordinate data in the feature semantic map, determine the coordinate data whose corresponding cosine similarity meets the target screening condition as the target coordinate data, and then determine the first range information based on the target coordinate data. The target coordinate data may include one or more coordinate data.

[0111] Exemplarily, after calculation, it is found that the cosine similarity between visual feature 1 corresponding to coordinate data 1 and the query image feature meets the target screening condition, and the cosine similarity between visual feature 2 corresponding to coordinate data 2 and the query image feature meets the target screening condition, so it is determined that the target coordinate data includes coordinate data 1 and coordinate data 2.

[0112] The target screening condition can be set as needed, for example, it can be set to that the corresponding cosine similarity is greater than or equal to a specific similarity threshold, or it can be set to that the corresponding cosine similarity ranks in the top N positions (sorted from large to small) among the cosine similarities of all coordinate data, where N is a preset positive integer.

[0113] In the case of determining multiple target coordinate data, if these target coordinate data all correspond to the same object, for example, in the above example, coordinate data 1 and coordinate data 2 both correspond to points on the surface of the sofa, then the position of the corresponding object can be determined as the center of the first range information, and the range within a preset distance around the center can be determined as the range indicated by the first range information (i.e. Figure 2 the range of the dotted circle).

[0114] When determining multiple target coordinate data, if these target coordinate data correspond to the same object respectively, for example, in the above example, coordinate data 1 corresponds to a point on the surface of the sofa, and coordinate data 2 corresponds to a point on the surface of the coffee table, then the midpoint of the multiple target coordinate data can be calculated, and the position of the midpoint can be determined as the center of the first range information, and the range within a preset distance around the center can be determined as the range indicated by the first range information.

[0115] If the query information includes text information, the server may obtain the first range information as follows:

[0116] The server uses a multimodal large model to process text information, obtain query text features, and then calculates the cosine similarity between the visual features corresponding to each coordinate data in the feature semantic map and the query text features, and determines the coordinate data whose cosine similarity meets the target screening conditions as the target coordinate data, and determines the first range information based on the target coordinate data.

[0117] The method of determining the target coordinate data and the method of determining the first range information based on the target coordinate data may refer to the aforementioned embodiment and will not be described in detail.

[0118] The advantage of obtaining the first range information in the above manner is that, in the related art, before repositioning only through images, it is necessary to manually semantically annotate various objects in the map, which is costly. In this embodiment, the first range information corresponding to the text information can be directly determined through a multimodal large model, without the need for manual semantic annotation in advance, thereby reducing the cost of applying the method of this embodiment.

[0119] The multimodal large model used in this embodiment can be an open source and trained multimodal large model in the relevant technical field, for example, various large language models; or, it can be a large model obtained by fine-tuning the aforementioned model using relevant image data and text data. Compared with directly training the entire model, this fine-tuning method can effectively reduce the amount of calculation and save the consumed computing resources.

[0120] If the query information includes image information and text information, the server can process the image information and text information in the above two ways respectively to obtain a range determined based on the image information and a range determined based on the text information, and then determine the intersection or union of the two ranges as the first range information.

[0121] In step S103, the second location information may be determined as follows:

[0122] The map data may include sensor data obtained by scanning at multiple positions in advance by other scanning devices. For example, within the range corresponding to the first range information, the map data may include sensor data 1 obtained by scanning at position 1 in advance, sensor data 2 obtained by scanning at position 2, and sensor data 3 obtained by scanning at position 3; the sensor data obtained by scanning at each position may include laser radar data obtained by scanning at the corresponding position using a laser radar, and image data obtained by photographing at the corresponding position using a depth camera;

[0123] After obtaining the target sensor data, the target sensor data can be compared with the sensor data of each position within the range indicated by the first range information, and the similarity of the target sensor data and the sensor data of each position within the range indicated by the first range information can be calculated, and the positions where the similarity is greater than a specific threshold, or the positions ranked in the top K positions after the similarity is sorted from large to small, are determined as candidate positions, where K is a preset positive integer;

[0124] If only one candidate location is determined, it can be determined that the candidate location is the location of the target device, and the location information representing the candidate location is the second location information of the target device;

[0125] If multiple candidate positions are determined, the midpoint between the multiple candidate positions can be calculated, and the position of the midpoint is determined as the position of the target device. The position information representing the position of the midpoint is the second position information of the target device.

[0126] In combination with the above example, assuming that it is determined that the similarity between sensor data 1 and target sensor data is greater than a specific threshold, it can be determined that position 1 obtained by scanning sensor data 1 is the position of the target device, and the position information corresponding to position 1 is the second position information;

[0127] Assuming that it is determined that the similarity between sensor data 1 and the target sensor data is greater than a specific threshold, and the similarity between sensor data 2 and the target sensor data is greater than a specific threshold, then position 1 and position 2 can be determined as alternative positions, and the midpoint between position 1 and position 2 is determined as the location of the target device, and the location information representing the location of the midpoint is the second location information of the target device.

[0128] The present application also provides a device for positioning a device, see Figure 4 , the device may include the following units.

[0129] An obtaining unit 401 is used to obtain query information related to the range where the target device is located;

[0130] A determination unit 402 is used to determine first range information of the target device according to the query information and the matching result of the pre-built feature semantic map, where the first range information represents the range where the target device is located;

[0131] The processing unit 403 is used to process the target sensor data of the target device according to the map data within the first range information to obtain the second position information of the target device, the second position information represents the position of the target device within the range indicated by the first range information, and the target sensor data includes at least one of the lidar data and the image data.

[0132] Optionally, the feature semantic map includes multiple coordinate data and visual features corresponding to the coordinate data, the coordinate data represents the position of an object in the environment where the target device is located, and the visual features corresponding to the coordinate data are visual features of the corresponding object.

[0133] Optionally, the device may further include a construction unit 404, configured to construct a feature semantic map in the following manner:

[0134] Obtain multiple coordinate data of the environment in which the target device is located, and multiple frames of key images obtained by photographing the environment;

[0135] Process the key image according to the multimodal large model to obtain the pixel features corresponding to each pixel in the key image;

[0136] For each coordinate data, according to the mapping relationship between the pixel points contained in the key image and the coordinate data, the pixel features of each pixel point corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

[0137] Optionally, when the construction unit 404 processes the key image according to the multimodal large model and obtains the pixel feature corresponding to each pixel point in the key image, it can be used to:

[0138] Extracting global image features and local image features of key images based on a multimodal large model;

[0139] The global features of the image are fused with the local features of the image where the pixel points in the key image are located to obtain the pixel features corresponding to the pixel points in the key image.

[0140] Optionally, when the construction unit 404 fuses the pixel features of each pixel point corresponding to the coordinate data to obtain the visual features corresponding to the coordinate data, it can be used for:

[0141] Filter out key area pixel features whose corresponding pixel points are located in the key area of ​​the image from multiple pixel features;

[0142] Filter out pixel features related to the occluded object from the pixel features of the key area to obtain filtered pixel features;

[0143] The filtered pixel features of the pixel points corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

[0144] Optionally, the query information includes at least one of image information and text information, the image information is obtained by photographing an object within a range of the target device, and the text information is obtained by a human-computer interaction module of the target device.

[0145] Optionally, when the determining unit 402 determines the first range information of the target device according to the query information and the matching result of the pre-built feature semantic map, it can be used to:

[0146] In the case where the query information includes image information, extracting global image features and key object features of the image information according to the multimodal macro model;

[0147] The query image features are obtained by fusing the global image features and the key object features;

[0148] The first range information of the target device is determined according to the matching result between the query image feature and the feature semantic map.

[0149] The working principle of the device for positioning equipment provided in this embodiment can be found in the relevant steps of the method for positioning equipment provided in any embodiment of the present application, and will not be repeated here.

[0150] Another embodiment of the application also provides an electronic device, such as Figure 5 As shown, specifically including:

[0151] One or more processors 501 .

[0152] The storage device 502 stores one or more programs.

[0153] When one or more programs are executed by one or more processors 501 , the one or more processors 501 implement a method for positioning a device as described in any one of the above embodiments.

[0154] The electronic device of this embodiment can be a target device. The target device can obtain query information and report the query information to a server used to run a multimodal large model. The server obtains first range information based on the query information and sends the first range information to the target device. After the target device obtains the first range information, it determines the second location information based on the map data and target sensor data in the first range information.

[0155] Another embodiment of the present application further provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for device positioning as described in any one of the above embodiments is implemented.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0157] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0158] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for positioning a device, characterized in that: include: Obtain query information related to the range of the target device; Determine first range information of the target device according to a matching result between the query information and a pre-built feature semantic map, where the first range information represents a range where the target device is located; The target sensor data of the target device is processed according to the map data within the first range information to obtain the second position information of the target device, wherein the second position information represents the position of the target device within the range indicated by the first range information, and the target sensor data includes at least one of lidar data and image data.

2. The method according to claim 1, characterized in that The feature semantic map includes a plurality of coordinate data and visual features corresponding to the coordinate data, the coordinate data represents the position of an object in the environment where the target device is located, and the visual features corresponding to the coordinate data are visual features of the corresponding object.

3. The method according to claim 2, characterized in that Methods for constructing the feature semantic map include: Obtaining multiple coordinate data of the environment in which the target device is located, and multiple frames of key images obtained by photographing the environment; Processing the key image according to the multimodal large model to obtain pixel features corresponding to each pixel in the key image; For each of the coordinate data, according to the mapping relationship between the pixel points contained in the key image and the coordinate data, the pixel features of each pixel point corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

4. The method according to claim 3, characterized in that Processing the key image according to the multimodal large model to obtain a pixel feature corresponding to each pixel in the key image includes: Extracting global image features and local image features of the key image according to the multimodal large model; The global image feature and the local image feature of the local area where the pixel point in the key image is located are fused to obtain the pixel feature corresponding to the pixel point in the key image.

5. The method according to claim 3, characterized in that: The step of fusing the pixel features of each pixel point corresponding to the coordinate data to obtain the visual features corresponding to the coordinate data includes: Filter out a key area pixel feature whose corresponding pixel point is located in a key area of ​​the image from the plurality of pixel features; Filtering out pixel features related to the occluded object from the pixel features of the key area to obtain filtered pixel features; The filtered pixel features of the pixel points corresponding to the coordinate data are fused to obtain the visual features corresponding to the coordinate data.

6. The method according to claim 1, characterized in that The query information includes at least one of image information and text information. The image information is obtained by the target device photographing an object within a range, and the text information is obtained by a human-computer interaction module of the target device.

7. The method according to claim 6, characterized in that The determining the first range information of the target device according to the matching result of the query information and the pre-built feature semantic map includes: In the case where the query information includes the image information, extracting global image features and key object features of the image information according to the multimodal macro model; fusing the global image features and the key object features to obtain query image features; The first range information of the target device is determined according to the matching result between the query image feature and the feature semantic map.

8. A device for positioning equipment, characterized in that: include: An obtaining unit, used to obtain query information related to the range where the target device is located; A determination unit, configured to determine first range information of the target device according to a matching result of the query information and a pre-built feature semantic map, wherein the first range information represents a range where the target device is located; A processing unit is used to process the target sensor data of the target device according to the map data within the first range information to obtain the second position information of the target device, wherein the second position information represents the position of the target device within the range indicated by the first range information, and the target sensor data includes at least one of lidar data and image data.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the device positioning method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for positioning a device as claimed in any one of claims 1 to 7 is implemented.

Images