Map data processing method, device and equipment and computer readable storage medium

Abstract

The invention provides a map data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a scene image and a depth image of a target scene, and performing two-dimensional information detection on the scene image to obtain object information and text information; performing pixel coordinate association processing on the object information, the text information and the depth image to obtain three-dimensional semantic information; determining an initial three-dimensional voxel map and an initial semantic map, and updating the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map; and extracting a semantic instance from the updated three-dimensional voxel map, and updating the initial semantic map based on the semantic instance to obtain an updated semantic map. According to the method and the device, the precision of the semantic map and the complex scene applicability can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a map data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] Semantic maps are an important research direction in the fields of computer vision and robotics. Semantic maps provide intelligent systems such as self-driving cars, mobile robots, augmented reality (AR), and virtual reality (VR) with the ability to understand and interpret the surrounding environment, enabling intelligent systems to make more advanced decisions and interact. With the continuous advancement of technology, the application of semantic maps will become more and more extensive, and will play an increasingly important role in the development of intelligent systems.

[0003] In the process of semantic map construction, related technologies only rely on object recognition and ignore text information. The current semantic map technology is insufficient in terms of diversified information processing and dynamic environment adaptability, resulting in low accuracy of semantic maps and low adaptability to complex scenarios, which in turn leads to poor perception and navigation capabilities of robots in complex environments. Summary of the invention

[0004] The embodiments of the present application provide a map data processing method, apparatus, device and computer-readable storage medium, which can improve the accuracy and adaptability of semantic maps to complex scenarios.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a map data processing method, the method comprising:

[0007] Acquire a scene image and a depth image of a target scene, and perform two-dimensional information detection on the scene image to obtain object information and text information;

[0008] Perform pixel coordinate association processing on the object information, the text information and the depth image to obtain three-dimensional semantic information;

[0009] Determining an initial three-dimensional voxel map and an initial semantic map, and updating the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map;

[0010] Semantic instances are extracted from the updated three-dimensional voxel map, and the initial semantic map is updated based on the semantic instances to obtain an updated semantic map.

[0011] The present application provides a map data processing device, including:

[0012] A first acquisition module is used to acquire a scene image and a depth image of a target scene, and perform two-dimensional information detection on the scene image to obtain object information and text information;

[0013] A second acquisition module is used to perform pixel coordinate association processing on the object information, the text information and the depth image to obtain three-dimensional semantic information;

[0014] A first updating module, configured to determine an initial three-dimensional voxel map and an initial semantic map, and update the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map;

[0015] The second updating module is used to extract semantic instances from the updated three-dimensional voxel map, and update the initial semantic map based on the semantic instances to obtain an updated semantic map.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions or computer programs;

[0018] The processor is used to implement the map data processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the map data processing method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the map data processing method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] By using the embodiment of the present application, the scene image and depth image of the target scene are obtained, and the scene image is subjected to two-dimensional information detection to obtain object information and text information, and then the object information, text information and depth image are subjected to pixel coordinate association processing to obtain three-dimensional semantic information, so as to realize the combination of depth information and scene image, ensure that the three-dimensional positioning of the object is more accurate, and then determine the initial three-dimensional voxel map and the initial semantic map, and update the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map, and then extract semantic instances from the updated three-dimensional voxel map, and update the initial semantic map based on the semantic instances to obtain an updated semantic map. In this way, by fusing the depth information, object information and text information of the image, a semantic map with multimodal information is constructed, the accuracy of the semantic map is improved, the limitations of relying on vision or depth perception are reduced, a more comprehensive environmental perception capability is provided for the robot, and the adaptability of the semantic map to complex scenes is improved through the dynamic update of the semantic map. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of an application mode of the map data processing method provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 3A is a first flow chart of the map data processing method provided in an embodiment of the present application;

[0026] Figure 3B is a second flow chart of the map data processing method provided in an embodiment of the present application;

[0027] Figure 3C is a third flow chart of the map data processing method provided in the embodiment of the present application;

[0028] Figure 3D is a fourth flow chart of the map data processing method provided in an embodiment of the present application;

[0029] Figure 3E is a fifth flow chart of the map data processing method provided in the embodiment of the present application;

[0030] Figure 4 It is a structural diagram of the map data processing system provided in an embodiment of the present application.

[0031] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0036] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) RGBD camera: It is a camera that captures color images and depth information simultaneously. It usually consists of a color camera and a depth sensor, and can output RGB images and corresponding depth images.

[0040] 2) Semantic map: It is an advanced map representation that contains not only spatial geometric information, but also semantic understanding of the objects and environment in the scene. In a semantic map, each element in the scene (such as buildings, roads, trees, vehicles, pedestrians, etc.) is assigned a specific semantic label that describes the nature, category, and function of the element.

[0041] 3) 3D point cloud: It is a data set consisting of a large number of discrete 3D space coordinate points. These points usually represent the shape and structure of 3D objects. Each point usually contains its position information in 3D space, and sometimes also includes other attributes such as color, intensity, normal direction, etc.

[0042] 4) Semantic classification: refers to the classification of objects in the scene into predefined categories. For example, objects in the scene are classified into categories such as "people", "cars", "trees", and "buildings".

[0043] The embodiments of the present application provide a map data processing method, apparatus, device and computer-readable storage medium, which can improve the accuracy and adaptability of semantic maps to complex scenarios.

[0044] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0045] See also Figure 1 , Figure 1 is a schematic diagram of an application mode of the map data processing method provided in an embodiment of the present application, for example, Figure 1 The server 200, the network 300 and the terminal 400 are involved. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0046] When updating the semantic map, the terminal 400 collects the scene image and depth image of the target scene, and sends the scene image and depth image of the target scene to the server 200. The server 200 receives the scene image and depth image of the target scene, and performs two-dimensional information detection on the scene image to obtain object information and text information; performs pixel coordinate association processing on the object information, text information and depth image to obtain three-dimensional semantic information; determines an initial three-dimensional voxel map and an initial semantic map, and updates the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map; extracts semantic instances from the updated three-dimensional voxel map, and updates the initial semantic map based on the semantic instances to obtain an updated semantic map, and returns the updated semantic map to the terminal 400. The terminal 400 receives the updated semantic map and uses the updated semantic map for navigation.

[0047] In some embodiments, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal 400 may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, vehicle-mounted terminal, aircraft, intelligent robot, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0048] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device may be a terminal or a server, Figure 2 The electronic device 500 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0049] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0050] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0051] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory. ， The volatile memory may be a random access memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiment of the present application is intended to include any suitable type of memory.

[0052] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0053] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0054] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0055] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A map data processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or plug-in, including the following software modules: a first acquisition module 4551, a second acquisition module 4552, a first update module 4553, and a second update module 4554. These modules are logical and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0056] The map data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0057] The following describes the map data processing method provided in the embodiment of the present application. For ease of understanding of the map data processing method provided in the embodiment of the present application, the following describes the application scenario of the map data processing method provided in the embodiment of the present application. When a robot navigates in a complex or dynamic environment, it needs to obtain a more comprehensive environmental perception capability through a semantic map. The map data processing method provided in the embodiment of the present application can improve the accuracy of the semantic map and its adaptability to complex scenarios.

[0058] As mentioned above, the electronic device implementing the map data processing method of the embodiment of the present application may be a terminal, a server, or a combination of the two. Next, the map data processing method provided by the embodiment of the present application is described by taking the electronic device as a server as an example. Figure 3A , Figure 3A is a first flow chart of the map data processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0059] In step 301, a scene image and a depth image of a target scene are acquired, and two-dimensional information detection is performed on the scene image to obtain object information and text information.

[0060] Here, the target scene is the current scene where the intelligent robot is located. The target scene can be a family life scene, a medical scene, a public service scene, etc., and the scene image can be a color image. The scene image and depth image of the target scene are obtained through the RGBD camera carried by the intelligent robot. Two-dimensional information detection is to perform object detection and text detection on the scene image, and input the scene image into the target detection network and the text detection network to obtain object information and text information respectively.

[0061] Here, object information includes object category, object location information and semantic image mask. Object category is to classify objects in the target scene into predefined categories. For example, object categories are divided into categories such as "people", "cars", "trees", and "buildings". Object location information is used to characterize the specific location of an object in space. Object location information can usually be represented by a rectangular bounding box or by the vertex coordinates of two diagonal vertices of the rectangular bounding box. Text information includes text content and text location information.

[0062] In step 302, pixel coordinate association processing is performed on the object information, text information, and depth image to obtain three-dimensional semantic information.

[0063] In some embodiments, see Figure 3B, Figure 3B It is a second flow chart of the map data processing method provided in the embodiment of the present application. Figure 3A Step 302 shown can be performed by Figure 3B Steps 3021 to 3026 are implemented as described in detail below.

[0064] In step 3021, point cloud computing is performed using the camera internal parameters and the depth image to obtain a three-dimensional point cloud.

[0065] Here, the camera internal parameters include focal length, principal point coordinates (the position of the optical center in the image coordinate system), and image distortion parameters. The depth image provides the distance information from each pixel to the camera. The depth image is usually represented in the form of a grayscale image, where the grayscale value of each pixel represents the distance from the pixel to the camera. For each pixel in the depth image, the position of the pixel in three-dimensional space can be calculated based on its depth value (distance) and the camera internal parameters to obtain a three-dimensional point cloud. A three-dimensional point cloud refers to a bunch of three-dimensional points containing three-dimensional position information, and each pixel corresponds to a three-dimensional point in the three-dimensional point cloud.

[0066] In step 3022, based on the semantic image mask, a first object point cloud is obtained from the three-dimensional point cloud, and a color mapping process is performed on the object category to obtain color information corresponding to the object category.

[0067] Here, a deep learning model is used to obtain the semantic image mask of the scene image. The semantic image mask is a binary image. The value of each pixel in the image is 0 or 1. The pixel value of the target scene area in the semantic image mask is 1, and the pixel value of the non-target scene area is 0. Through the image semantic mask, the three-dimensional spatial coordinates of each pixel point belonging to the object are obtained from the three-dimensional point cloud to obtain the first object point cloud. The object category can be represented by a discrete value, for example: 0 to 80. Using the Jet color mapping method, the discrete value of the object category is mapped to the corresponding RGB color value to obtain the RGB color information corresponding to the object category, where the R, G, and B color values ​​are all in the range of 0-255.

[0068] In step 3023, color information corresponding to the object category is assigned to the first object point cloud to obtain a second object point cloud.

[0069] Here, for each 3D point in the first object point cloud, according to the object category to which it belongs, the color information corresponding to the object category is obtained from the color map, and the color information is assigned to the corresponding 3D point through the point cloud visualization software (such as PCL, Open3D, Mayavi, etc.), so as to obtain the second object point cloud. Each 3D point in the second object point cloud corresponds to a 3D space coordinate and a color value.

[0070] In step 3024, based on the text position information, a first text point cloud is obtained from the three-dimensional point cloud, and a preset color value corresponding to the text category is obtained.

[0071] Here, two-dimensional information detection is performed on the scene image to generate a rectangular bounding box of the text, and the vertex coordinates of the two diagonal vertices of the rectangular bounding box of the text can be used as the text position information. The text position information is mapped to the three-dimensional point cloud through the internal parameters of the camera (focal length, principal point coordinates, etc.) and the camera's posture (position and direction), and the three-dimensional point cloud is cropped according to the text position information, and only the three-dimensional points related to the text area are retained to obtain the first text point cloud. The color value corresponding to the text category is pre-set. For example, the green color value (RGB value is 0, 255, 0) can be used as the preset color value corresponding to the text category.

[0072] In step 3025, a preset color value is assigned to the first text point cloud to obtain a second text point cloud.

[0073] Here, for each 3D point in the first text point cloud, a preset color value is assigned to the corresponding 3D point through point cloud visualization software (such as PCL, Open3D, Mayavi, etc.) to obtain a second text point cloud. Each 3D point in the second text point cloud corresponds to a 3D space coordinate and a preset color value.

[0074] In step 3026, three-dimensional semantic information is generated based on the second object point cloud, the second text point cloud, and the text content.

[0075] Here, the three-dimensional semantic information includes semantic point cloud information and a three-dimensional text label array.

[0076] In some embodiments, three-dimensional semantic information is generated based on the second object point cloud, the second text point cloud and the text content by performing the following steps: combining the second object point cloud and the second text point cloud to obtain semantic point cloud information; obtaining the three-dimensional text coordinates of the text content in the camera coordinate system from the second text point cloud; and combining the three-dimensional text coordinates and the text content to obtain a three-dimensional text marker array.

[0077] Here, the semantic point cloud information is obtained by combining the second object point cloud and the second text point cloud, and is used to characterize the object information and text information in the three-dimensional point cloud. Optical Character Recognition (OCR) technology is used to identify the text content in the second text point cloud, and the three-dimensional text coordinates of the text content in the camera coordinate system are determined. The three-dimensional text tag array includes multiple three-dimensional text tags, each of which is composed of three-dimensional text coordinates and text content. The text content is represented by a character string. For example, the three-dimensional text coordinates of the three-dimensional text tag T1 are (5.0, 2.5, 3.1), and the text content is "Alpha Conference Room".

[0078] In an embodiment of the present application, object information, text information and depth image are processed by pixel coordinate association to obtain three-dimensional semantic information, and the depth information can be combined with the scene image to improve the accuracy of three-dimensional positioning of the object.

[0079] Continue to refer Figure 3A In step 303, an initial three-dimensional voxel map and an initial semantic map are determined, and the initial three-dimensional voxel map is updated based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map.

[0080] Here, the initial three-dimensional voxel map is a three-dimensional voxel map to be updated, and a three-dimensional sensor such as a lidar is used to collect three-dimensional data of the target scene. The three-dimensional data is converted into an initial three-dimensional voxel map using a truncated signed distance function (TSDF). The initial three-dimensional voxel map consists of multiple voxels of fixed size, each of which contains its three-dimensional position coordinates in the world coordinate system, truncated signed distance, semantic category probability, and text content. The truncated signed distance represents the distance between the voxel and the nearest obstacle, and the semantic category probability is the probability of the category to which the voxel belongs. The initial semantic map is a semantic map to be updated. The semantic map is a map that includes spatial geometric information and semantic understanding of objects and environments in the target scene. The initial semantic map of the target scene is constructed through the simultaneous localization and mapping (SLAM) technology.

[0081] In some embodiments, see Figure 3C , Figure 3C This is a third flow chart of the map data processing method provided in the embodiment of the present application. Figure 3A In step 303 shown in the figure, "updating the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map" can be achieved by Figure 3C Steps 3031 to 3033 are implemented as described in detail below.

[0082] In step 3031, the camera center position is determined, and based on the camera center position, a voxel set to be updated is determined from the initial three-dimensional voxel map.

[0083] Here, the first distance between each voxel to be updated in the set of voxels to be updated and the center position of the camera is less than or equal to the first preset distance. The first distance is the distance between each voxel to be updated and the center position of the camera, and the first preset distance is a preset distance. The center position of the camera is the position of the robot in the target scene, and the ray direction between the center position of the camera and each voxel in the initial three-dimensional voxel map is determined. The voxels located in the ray direction that are less than or equal to the first preset distance are determined as voxels to be updated, and multiple voxels to be updated are combined into a set of voxels to be updated. For example, the first preset distance is 6 meters, and the voxels whose first distance from the center position of the camera is less than or equal to 6 meters are determined as voxels to be updated, and multiple voxels to be updated within 6 meters are combined into a set of voxels to be updated.

[0084] In step 3032, each voxel to be updated is updated based on the semantic point cloud information in the three-dimensional semantic information to obtain each updated voxel.

[0085] Here, based on the semantic point cloud information in the three-dimensional semantic information, the truncated signed distance, weight and semantic category probability corresponding to each voxel to be updated are updated to obtain each updated voxel.

[0086] In some embodiments, based on the semantic point cloud information in the three-dimensional semantic information, each voxel to be updated is updated to obtain each updated voxel, which can be achieved by performing the following steps: for each voxel to be updated, the first truncated signed distance, the first weight and the first semantic category probability corresponding to the voxel to be updated are obtained; the updated truncated signed distance and the second weight corresponding to the voxel to be updated are obtained from the semantic point cloud information; the first truncated signed distance and the updated truncated signed distance are merged with the first weight and the second weight to obtain the second truncated signed distance; the historical semantic category of the voxel to be updated is obtained, and the maximum likelihood estimation is performed on the historical semantic category to obtain the second semantic category probability; the first truncated signed distance corresponding to the voxel to be updated is replaced by the second truncated signed distance, and the first semantic category probability is replaced by the second semantic category probability to obtain the updated voxel.

[0087] Here, in the initial three-dimensional voxel map, the first truncated signed distance is the truncated signed distance corresponding to each voxel to be updated, the first weight is the weight corresponding to each voxel to be updated, which is used to characterize the probability or degree of occupation of the voxel to be updated, and the first semantic category probability is the probability of the category to which each voxel to be updated belongs. For each voxel to be updated, the updated truncated signed distance and the second weight corresponding to the voxel to be updated are obtained from the semantic point cloud information, and the second weight is the updated weight corresponding to each voxel to be updated. Using the first weight and the second weight, the first truncated signed distance and the updated truncated signed distance are weighted averaged to obtain the target truncated signed distance corresponding to the voxel to be updated, that is, the second truncated signed distance. For example, the calculation formula for the second truncated signed distance is as follows:

[0088]

[0089] Among them, x represents the center position of the voxel to be updated, p represents the position of a point in the semantic 3D point cloud, s represents the center position of the camera, and W i represents the first weight, w represents the second weight, W i+1 represents the updated weight, W max represents the maximum value of the weight, D i represents the first truncated symbol distance, d represents the updated truncated symbol distance, D i+1 Represents the second truncated signed distance.

[0090] The historical semantic category is the semantic category observed in the history of the voxel to be updated. The probability model such as the naive Bayes classifier and the logistic regression model is used to perform maximum likelihood estimation on the historical semantic category to obtain the target semantic category probability corresponding to the voxel to be updated, that is, the second semantic category probability. In the initial three-dimensional voxel map, for each voxel to be updated, the first truncated signed distance corresponding to the voxel to be updated is replaced with the second truncated signed distance, and the first semantic category probability corresponding to the voxel to be updated is replaced with the second semantic category probability to obtain each updated voxel.

[0091] In step 3033, the voxels to be updated in the initial three-dimensional voxel map are replaced with the corresponding updated voxels to obtain an updated three-dimensional voxel map.

[0092] Here, for each voxel to be updated in the initial three-dimensional voxel map, the voxel to be updated is replaced with the corresponding updated voxel to obtain an updated three-dimensional voxel map.

[0093] In an embodiment of the present application, each voxel to be updated is updated based on the semantic point cloud information in the three-dimensional semantic information, and the voxels to be updated in the initial three-dimensional voxel map are replaced with the corresponding updated voxels to obtain an updated three-dimensional voxel map. This realizes timely updating of the initial three-dimensional voxel map based on the newly acquired three-dimensional semantic point cloud information, thereby improving the accuracy of subsequent extraction of semantic instances from the updated three-dimensional voxel map.

[0094] Continue to refer Figure 3A In step 304, semantic instances are extracted from the updated three-dimensional voxel map, and the initial semantic map is updated based on the semantic instances to obtain an updated semantic map.

[0095] Here, semantic instances include object instances and text instances. Object instances include object identifiers, object semantic categories, mesh grids associated with the object, the central 3D coordinates of the object, and the 3D bounding box of the object. Text instances include text identifiers, mesh grids associated with the text, the central 3D coordinates of the text, and text content.

[0096] In some embodiments, see Figure 3D , Figure 3D This is a fourth flow chart of the map data processing method provided in the embodiment of the present application. Figure 3A In step 304 shown in the figure, "extracting semantic instances from the updated three-dimensional voxel map" can be done by Figure 3D Steps 3041 to 3045 are implemented as described in detail below.

[0097] In step 3041, mesh data is extracted from the updated three-dimensional voxel map.

[0098] Here, the isosurface extraction Marching Cubes algorithm is used to extract grid data from the updated three-dimensional voxel map. The grid data is a Mesh grid, which is composed of vertices, edges and triangular faces. The grid data includes at least multiple vertices and the semantic category of each vertex.

[0099] In step 3042, at least one first target vertex is determined from the plurality of vertices based on a second distance between each vertex and the camera center position.

[0100] Here, the second distance is the distance between each vertex in the mesh data and the camera center position. At least one first target vertex is determined from multiple vertices. The second distance between the first target vertex and the camera center position is less than or equal to a second preset distance. The second preset distance is a distance within a fixed radius based on the camera center position.

[0101] In step 3043, clustering is performed on at least one first target vertex to obtain a first cluster, and the first cluster is determined as a first object instance.

[0102] Here, at least one first target vertex is combined to obtain a first cluster, where the first cluster includes a plurality of first target vertices and a semantic category of each first target vertex.

[0103] In some embodiments, clustering at least one first target vertex to obtain a first cluster can be achieved by performing the following steps: determining a third distance between different first target vertices, and clustering at least one first target vertex based on the third distance to obtain a first cluster.

[0104] Here, the Euclidean clustering method is used to calculate the Euclidean distance between each first target vertex and other first target vertices, that is, the third distance, and the other first target vertices whose third distance is less than the set threshold are combined with the first target vertex to obtain the first cluster.

[0105] According to the semantic category of each first target vertex in the first cluster, the number of first target vertices corresponding to different semantic categories is determined, and the semantic category corresponding to the largest number of first target vertices is determined as the semantic category of the first cluster. The first object instance includes the first cluster and the semantic category of the first cluster.

[0106] For example, the semantic categories of each first target vertex in the first cluster include the semantic category "chair" and the semantic category "sofa". The number of first target vertices corresponding to the semantic category "chair" is 10, the number of first target vertices corresponding to the semantic category "sofa" is 15, and the semantic category corresponding to the largest number of first target vertices is the semantic category "sofa". The semantic category "sofa" is determined as the semantic category of the first cluster, and the first cluster and the semantic category "sofa" of the first cluster are determined as the first object instance.

[0107] In step 3044, at least one second target vertex whose semantic category is the text category is determined from the multiple vertices, and clustering is performed on the at least one second target vertex to obtain a second cluster.

[0108] Here, at least one second target vertex with a semantic category of text is determined from multiple vertices of the grid data, and the at least one second target vertex is clustered to obtain a second cluster. The second cluster includes multiple second target vertices and the text category of each second target vertex.

[0109] In some embodiments, clustering is performed on at least one second target vertex to obtain a second cluster cluster, which can be achieved by executing the following steps: obtaining the three-dimensional coordinates of the text content in the camera coordinate system from the three-dimensional text mark array in the three-dimensional semantic information; determining a fourth distance between each second target vertex and the three-dimensional coordinate, and based on the fourth distance, clustering is performed on at least one second target vertex to obtain a second cluster cluster.

[0110] Here, for each 3D text mark in the 3D text mark array, the 3D coordinates of the text content of the 3D text mark in the camera coordinate system are determined. The distance between each second target vertex and the 3D coordinate, i.e., the fourth distance, is calculated, and the second target vertices whose fourth distance is less than a set threshold and whose semantic category is the text category are combined to obtain a second cluster.

[0111] In step 3045 , the second cluster is determined as the first text instance.

[0112] Here, the second clustering cluster and the text category of each second target vertex in the second clustering cluster are combined into the first text instance.

[0113] In an embodiment of the present application, at least one first target vertex is clustered to obtain a first cluster, and the first cluster is determined as a first object instance, and at least one second target vertex is clustered to obtain a second cluster, and the second cluster is determined as a first text instance, thereby realizing clustering based on target vertices that meet preset distances and text categories, and improving the accuracy of determining object instances and text instances.

[0114] In some embodiments, see Figure 3E , Figure 3E This is a fifth flow chart of the map data processing method provided in the embodiment of the present application. Figure 3A In step 304 shown in the figure, "update the initial semantic map based on the semantic instance to obtain an updated semantic map" can be achieved by Figure 3E Steps 310 to 314 are implemented as described in detail below.

[0115] In step 310, when the initial semantic map includes a second object instance that satisfies a first proximity condition with the first object instance, an overlap rate between the first object instance and the second object instance is determined.

[0116] Here, the second object instance is an object instance existing in the initial semantic map. When the initial semantic map includes the second object instance that satisfies the first proximity condition with the first object instance, the volume of the intersection area and the volume of the union area of ​​the two bounding boxes in the three-dimensional space are calculated by using the three-dimensional bounding box of the object in the first object instance and the three-dimensional bounding box of the object in the second object instance, and the ratio of the volume of the intersection area to the volume of the union area is determined as the overlap rate of the first object instance and the second object instance.

[0117] In some embodiments, determining a second object instance that satisfies a first proximity condition with a first object instance can be achieved by performing the following steps: determining a candidate object instance from an initial semantic map, the candidate object instance having the same semantic category as the first object instance; determining a fifth distance between the candidate object instance and the first object instance; and when the fifth distance is less than a third preset threshold, determining the candidate object instance as a second object instance that satisfies the first proximity condition with the first object instance.

[0118] Here, according to the semantic category of the first object instance, a candidate object instance with the same semantic category as the first object instance is determined from the initial semantic map. According to the three-dimensional coordinates of the center point of the candidate object instance and the three-dimensional coordinates of the center point of the first object instance, the distance between the candidate object instance and the first object instance, i.e., the fifth distance, is calculated. The third preset threshold is a pre-set distance threshold that satisfies the first proximity condition. When the fifth distance is less than the third preset threshold, the candidate object instance is determined as a second object instance that satisfies the first proximity condition with the first object instance.

[0119] In step 311, when the overlap rate is greater than a first preset threshold, the first object instance and the second object instance are merged to obtain a merged object instance.

[0120] Here, the first preset threshold is a preset overlap rate threshold. When the overlap rate between the first object instance and the second object instance is greater than the first preset threshold, the Mesh mesh associated with the first object instance and the Mesh mesh associated with the second object instance are merged to obtain a merged mesh vertex set. Based on the merged mesh vertex set, the central three-dimensional coordinates and the three-dimensional bounding box of the object instance are calculated to obtain a merged object instance.

[0121] In step 312, when the initial semantic map includes a second text instance that satisfies a second proximity condition with the first text instance, a ratio of identical characters between the first text instance and the second text instance is determined.

[0122] Here, the second text instance is a text instance existing in the initial semantic map. When the initial semantic map includes a second text instance that satisfies the second proximity condition with the first text instance, based on the text content in the first text instance and the text content in the second text instance, the number of identical characters of the first text instance and the second text instance is determined, and the first number of characters of the text content in the first text instance and the second number of characters of the text content in the second text instance are determined, the maximum value of the first number of characters and the second number of characters is determined as the maximum number of characters, and the ratio of the number of identical characters to the maximum number of characters is determined as the identical character ratio.

[0123] In some embodiments, determining a second text instance that satisfies a second proximity condition with a first text instance can be achieved by performing the following steps: determining a candidate text instance from an initial semantic map, and determining a sixth distance between the candidate text instance and the first text instance; when the sixth distance is less than a fourth preset threshold, determining the candidate text instance as a second text instance that satisfies a second proximity condition with the first text instance.

[0124] Here, the distance between the candidate text instance and the first text instance, i.e., the sixth distance, is calculated based on the central three-dimensional coordinates of the candidate text instance and the central three-dimensional coordinates of the first text instance. The fourth preset threshold is a preset distance threshold that satisfies the second proximity condition. When the sixth distance is less than the fourth preset threshold, the candidate text instance is determined to be the second text instance that satisfies the second proximity condition with the first text instance.

[0125] In step 313, when the proportion of identical characters is greater than a second preset threshold, the first text instance and the second text instance are merged to obtain a merged text instance.

[0126] Here, the second preset threshold is a preset threshold of the ratio of identical characters. When the ratio of identical characters between the first text instance and the second text instance is greater than the second preset threshold, the Mesh mesh associated in the first text instance and the Mesh mesh associated in the second text instance are merged to obtain a merged set of mesh vertices. Based on the merged set of mesh vertices, the central three-dimensional coordinates of the text instance are calculated to obtain a merged text instance. The merged text instance will record all observed text contents, and the text content with the most observations will be used as the text content of the merged text instance.

[0127] In step 314, the second object instance in the initial semantic map is replaced with the merged object instance, and the second text instance is replaced with the merged text instance to obtain an updated semantic map.

[0128] Here, the initial semantic map is updated based on the merged object instance and the merged text instance. For the second object instance in the initial semantic map, the second object instance is replaced with the merged object instance. For the second text instance in the initial semantic map, the second text instance is replaced with the merged text instance to obtain an updated semantic map.

[0129] In an embodiment of the present application, the second object instance in the initial semantic map is replaced with the merged object instance, and the second text instance is replaced with the merged text instance to obtain an updated semantic map. The semantic map can be dynamically updated through the newly acquired object instances and text instances, thereby improving the accuracy of the semantic map and its adaptability to complex scenarios.

[0130] In some embodiments, the map data processing method provided in the embodiments of the present application can be applied in the field of robot navigation. During navigation, the intelligent robot analyzes obstacles, traversable areas, dangerous areas and other information in the semantic map based on the updated semantic map to determine the optimal path planning and obstacle avoidance strategy.

[0131] The server obtains the scene image and depth image of the target scene, and performs two-dimensional information detection on the scene image to obtain object information and text information, and then performs pixel coordinate association processing on the object information, text information and depth image to obtain three-dimensional semantic information, realizing the combination of depth information and scene image to ensure more accurate three-dimensional positioning of the object, and then determines the initial three-dimensional voxel map and initial semantic map, and updates the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map, and then extracts semantic instances from the updated three-dimensional voxel map, and updates the initial semantic map based on the semantic instances to obtain an updated semantic map. In this way, by fusing the depth information, object information and text information of the image, a semantic map with multimodal information is constructed, the accuracy of the semantic map is improved, and a more comprehensive environmental perception capability is provided for the robot, and the adaptability of the semantic map to complex scenes is improved through the dynamic update of the semantic map.

[0132] The following describes an exemplary application of the map data processing method provided in an embodiment of the present application in a robot navigation scenario.

[0133] With the development of robotics, semantic maps are widely used in the research of robot perception and navigation. Traditional semantic map construction methods usually only combine spatial information and object classification, but in complex environments, text information (such as signs, labels, etc.) is also an important semantic element that can name spatial areas and provide rich navigation information.

[0134] In the process of semantic map construction, related technologies only rely on object recognition and ignore text information, resulting in the current semantic map technology's insufficient capabilities in diversified information processing and dynamic environment adaptability, which in turn leads to the robot's poor perception and navigation capabilities in complex environments.

[0135] In view of the problems existing in the related art, the embodiment of the present application proposes a map data processing method, which includes the following improvements compared with the related art:

[0136] 1) Using an RGBD camera to obtain a color image and a depth image, obtaining object information (the above-mentioned object information) and text information of the color image through target detection and text detection, and generating 3D semantic information by combining the object information, text information and depth image. The 3D semantic information includes a 3D point cloud with semantic information and a 3D text label array;

[0137] 2) Process the 3D point cloud with semantic information by voxel fusion, use the 3D semantic information and the robot's positioning information to update the voxel map, extract object instances (the above-mentioned object instances) and text instances from the voxel map by clustering, and thus construct a semantic map with multimodal information based on object instances and text instances.

[0138] In the embodiment of the present application, through the image data and depth information provided by the RGBD camera, object information and text information can be extracted and a semantic three-dimensional point cloud can be generated, which has significant advantages in semantic recognition and three-dimensional reconstruction. The extracted object information and text information can be integrated into the same semantic map, which can greatly enhance the robot's perception ability and navigation task execution efficiency in complex environments.

[0139] For example, the robot obtains color images and depth images through the RGBD camera it carries, and inputs the color images into the target detection network and the text detection network to obtain object information and text information respectively. Then, the object information and text information are associated with the pixel coordinates of the depth image and converted into a three-dimensional point cloud with semantic information and a three-dimensional text tag array. Each text tag in the three-dimensional text tag array contains text content and text three-dimensional coordinates. The semantic three-dimensional point cloud, three-dimensional text tag array, and robot positioning information are input into the mapping module, and the current voxel map is updated by voxel fusion. Then, object instances and text instances are extracted from the updated voxel map by clustering, thereby constructing and updating the semantic map.

[0140] Example, reference Figure 4 , Figure 4 Schematic diagram of the structure of the map data processing system provided in the embodiment of the present application. Figure 4 The map data processing system in the embodiment of the present application is used to explain the map data processing method provided in the embodiment of the present application.

[0141] The map data processing system uses the target detection network 502 and the text detection network 503 to perform two-dimensional semantic information detection, and respectively extracts the object information 504 and text information 505 of the color image 501. Among them, the object information 504 includes the category of the semantic object, the object rectangular bounding box and the image mask (the image mask refers to a binary image, and the value of each pixel in the image is 0 or 1. If the pixel is located on the semantic object, it is 1, and if the pixel is not located on the semantic object, it is 0. The coordinates of each pixel belonging to the semantic object are obtained through the image mask), and the text information 505 includes the string of text content and the text rectangular bounding box.

[0142] In the point cloud computing module 507, the camera intrinsic parameters and the depth image 506 are used to generate a three-dimensional point cloud 508, and the object information 504 and the text information 505 are associated with the three-dimensional point cloud 508, so as to obtain three-dimensional semantic information 509, which includes a semantic three-dimensional point cloud 510 and a three-dimensional text tag array 511. Specifically, for the object information, the system uses the image mask to obtain the corresponding pixel coordinate set of the object information, and can convert the pixel coordinate value of the object information into a three-dimensional coordinate value in combination with the depth image, so as to obtain the three-dimensional point cloud of the object information. A three-dimensional point cloud refers to a bunch of points containing three-dimensional position information, and each pixel only corresponds to one three-dimensional point. Since the three-dimensional point cloud can store the color information of each pixel of the object in addition to the three-dimensional position information of the object, in order to express the category information of the semantic object in the three-dimensional point cloud (represented by discrete integers), the Jet color mapping method is used to map the discrete semantic object category value (such as 0-80) to the corresponding RGB color value, where the range of the R, G, and B color values ​​is 0-255. After obtaining the RGB color corresponding to the category of the semantic object, the corresponding RGB color value can be assigned to the three-dimensional point cloud of the object information.

[0143] For text information, the system uses the text rectangular bounding box to obtain the corresponding pixel coordinate set of the text information, and uses a specific single color to represent the corresponding pixels of the text information, and assigns the specific single color value to the three-dimensional point cloud of the text information. For example, the green color value (RGB value is 0, 255, 0) is assigned to the three-dimensional point cloud of the text information, thereby expressing that the three-dimensional point cloud belongs to the text information. If only the semantic three-dimensional point cloud is used to construct the semantic map, the text content in the text information will be missing. Therefore, the system also uses the point cloud to calculate the three-dimensional coordinates of the text information in the camera coordinate system, and forms a three-dimensional text tag array with the content string in the text information. Each three-dimensional text tag in the three-dimensional text tag array includes the three-dimensional coordinate value (X, Y, Z) of the text tag, and the content string of the text tag. For example, the three-dimensional text tag T1 includes the three-dimensional coordinate value (5.0, 2.5, 3.1), and the content string "Alpha Conference Room".

[0144] The system uses the Truncated Signed Distance Function (TSDF) to express the three-dimensional voxel map. The three-dimensional voxel map is composed of multiple voxels of fixed size. Each voxel includes its three-dimensional position coordinates in the world coordinate system, the truncated signed distance (the distance between the voxel and the nearest obstacle), the probability of the semantic category (since the semantic object detection network can detect 80 object categories, plus an additional non-object category, there are a total of 81 categories. The probability value of each voxel belonging to the 81 categories is maintained, and the category with the highest probability value is taken as the category of the voxel) and text content (empty if it is a non-text category).

[0145] Continue to refer Figure 4, the map data processing system combines the semantic three-dimensional point cloud 510, the three-dimensional text mark array 511 and the camera positioning information 512, uses the voxel fusion method to update the voxel map, performs voxel map construction and grid extraction 513, and obtains a local grid 514. In some embodiments, the camera positioning information (corresponding to the camera center position in the above embodiment) can express the current position of the robot in the scene, and selects the local voxel map within a fixed range (such as 6 meters) of the current robot position through the camera positioning information, and determines the voxels in the local voxel map as the voxel set to be updated. Use the newly acquired semantic three-dimensional point cloud to update the voxel set to be updated. For example, for each point in the semantic three-dimensional point cloud, the ray direction between the camera center and the point can be calculated, and the voxel set located on the ray is determined as the voxel set to be updated, so as to update the truncated signed distance and semantic category probability of each voxel in the voxel set according to the semantic three-dimensional point cloud. In the voxel map, the first truncated signed distance, the first weight and the first semantic category probability of each voxel are maintained. According to each point in the semantic 3D point cloud, the updated truncated signed distance and the second weight are calculated. For example, the calculation formula for the updated truncated signed distance is as follows:

[0146]

[0147] Among them, x represents the center position of the voxel to be updated, p represents the position of a point in the semantic 3D point cloud, s represents the center position of the camera, and W i represents the first weight, w represents the second weight, W i+1 represents the updated weight, W max represents the maximum value of the weight, D i represents the first truncated symbol distance, d represents the updated truncated symbol distance, D i+1 Represents the second truncated signed distance.

[0148] For semantic category probability, the maximum likelihood estimation is used to update the semantic category probability based on the number of semantic categories observed in the voxel history. The system uses the Marching Cubes algorithm to extract the Mesh grid (the grid consists of vertices, edges, and triangles) from the 3D voxel map, and obtains the semantic category with the highest probability through the probability of the semantic category in the voxel, and uses it as the semantic category of the Mesh grid vertex.

[0149] Continue to refer Figure 4, based on the local grid 514, semantic instance extraction 515 is performed to obtain a new semantic instance 516. Semantic instances include object instances and text instances. For object instances, a local area within a fixed radius can be selected according to the position information of the robot, and then a set of Mesh grid vertices in this local area is obtained to extract object instances within the local range around the robot. The Euclidean clustering method is adopted, that is, a vertex in the vertex set is randomly selected, and the similar vertices with the same semantic label and the distance to the selected vertex is judged to be less than the set threshold according to the Euclidean distance between the vertices, and the selected vertex and the similar vertices are combined into a cluster. Repeat the above steps until all vertices are divided into corresponding clusters. For each cluster, the number of vertices belonging to each semantic category label is counted. For example, the number of vertices belonging to the chair category label is 10, and the number of vertices belonging to the sofa category label is 15. The semantic category label corresponding to the largest number of vertices is used as the semantic category of the cluster, so that the cluster is determined as an object instance. Each object instance contains a semantic object identifier, a semantic object category, an associated Mesh mesh, the central 3D coordinates in the world coordinate system, and the 3D bounding box of the object.

[0150] For text instances, each 3D text tag in the 3D text tag array is processed in turn, and the 3D coordinate value in the 3D text tag is used. Then, the mesh vertices near the coordinates and belonging to the text category are obtained according to the 3D coordinate value, that is, whether the distance between the vertex and the coordinate meets the threshold and whether the category attribute of the vertex is the text category is judged. The Mesh mesh vertices belonging to the text category near the 3D text tag are regarded as clusters and used as a text instance. Each text instance contains a text identifier, an associated Mesh mesh, a central 3D coordinate in the world coordinate system, and a text content string.

[0151] Continue to refer Figure 4, based on the new semantic instance 516, the semantic map is updated 517 to obtain a multimodal semantic map 518. After extracting the new semantic instance, the system will associate the new semantic instance with the semantic instance in the current semantic map to update the semantic map. For a new object instance, the system uses the object semantic category and the center three-dimensional coordinates in the object instance to determine whether the new object instance is similar to the object instance with the same object semantic category in the semantic map, and uses the three-dimensional bounding box of the new object instance and the existing old object instance to calculate the overlap rate of the two object instances. When the overlap rate exceeds a certain value, it is regarded as the same object instance, and the new object instance is merged into the old object instance, and the associated Mesh mesh sets in the new object instance and the old object instance are merged. Specifically, each object instance will be associated with a set of mesh vertices in the Mesh map, the mesh vertex sets of the new object instance and the old object instance will be merged, the average coordinate value of the merged mesh vertex set will be recalculated, the average coordinate value will be used as the central three-dimensional coordinate of the merged object instance, and the minimum bounding box of the merged mesh vertex set will be calculated, and the minimum bounding box will be used as the three-dimensional bounding box of the merged object instance.

[0152] For a new text instance, the system uses the central three-dimensional coordinates in the text instance to determine whether there is an old text instance similar to the new text instance, and compares the text content strings of the new text instance and the similar old text instance. If the ratio of the number of identical characters to the total number of characters (the total number of characters is the maximum number of strings in the new text instance and the similar old text instance) exceeds a certain value, they are regarded as the same text instance, and the new text instance is merged into the old text instance to obtain a merged text instance, and the associated Mesh grid, central three-dimensional coordinates and text content string in the merged text instance are updated at the same time. The merged text instance will record the content string set of all observed new text instances, and the string with the largest number of observations will be used as the text content string of the merged text instance.

[0153] The semantic map generated in the embodiment of the present application contains multimodal information, which enables the robot to understand richer contextual information and helps the robot to obtain a more comprehensive environmental perception ability when navigating in a complex or dynamic environment. By combining object information and text information, the robot can navigate on the basis of higher-level semantic information, thereby improving the adaptability to complex scenes, reducing the limitations of relying on traditional vision or depth perception, and improving the robustness and reliability of the robot's navigation in different environments. In addition, by integrating text information into the semantic map, the robot can better understand human language instructions. For example, a user can use voice or text instructions to ask the robot to "go to the kitchen" or "find an exit", and the robot can accurately complete the instructions by recognizing the text logo in the map, improving the naturalness and convenience of the interaction.

[0154] In the above-mentioned robot navigation application scenario, the depth image obtained by the RGBD camera is combined with the color image to ensure more accurate three-dimensional positioning of the object, and the environmental context information of the semantic map is enriched and diverse by adding text information. By integrating image data, object information and text information, a semantic map with multimodal information is constructed to provide the robot with a more comprehensive environmental perception capability. Through the dynamic update and optimization mechanism of the semantic map, the real-time and adaptability of the semantic map are ensured, and the robot can continuously obtain new environmental information during movement, thereby adapting to the ever-changing environment.

[0155] The following is a description of an exemplary structure of a map data processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2 As shown, the software modules stored in the map data processing device 455 of the memory 450 may include: a first determination module 4551, used to obtain a scene image and a depth image of the target scene, and perform two-dimensional information detection on the scene image to obtain object information and text information; a second determination module 4552, used to perform pixel coordinate association processing on the object information, text information and depth image to obtain three-dimensional semantic information; a first update module 4553, used to determine an initial three-dimensional voxel map and an initial semantic map, and update the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map; a second update module 4554, used to extract semantic instances from the updated three-dimensional voxel map, and update the initial semantic map based on the semantic instances to obtain an updated semantic map.

[0156] In some embodiments, the object information includes object categories and semantic image masks, and the text information includes text content and text location information. The second determination module 4552 is also used to use the camera internal parameters and the depth image to perform point cloud computing processing to obtain a three-dimensional point cloud; based on the semantic image mask, obtain the first object point cloud from the three-dimensional point cloud, perform color mapping processing on the object category, and obtain color information corresponding to the object category; assign the color information corresponding to the object category to the first object point cloud to obtain a second object point cloud; based on the text location information, obtain the first text point cloud from the three-dimensional point cloud, and obtain the preset color value corresponding to the text category; assign the preset color value to the first text point cloud to obtain a second text point cloud; generate three-dimensional semantic information based on the second object point cloud, the second text point cloud and the text content.

[0157] In some embodiments, the three-dimensional semantic information includes semantic point cloud information and a three-dimensional text marker array. The second determination module 4552 is also used to combine the second object point cloud and the second text point cloud to obtain semantic point cloud information; obtain the three-dimensional text coordinates of the text content in the camera coordinate system from the second text point cloud; and combine the three-dimensional text coordinates and text content to obtain a three-dimensional text marker array.

[0158] In some embodiments, the first update module 4553 is also used to determine the camera center position, and based on the camera center position, determine a set of voxels to be updated from the initial three-dimensional voxel map, wherein a first distance between each voxel to be updated in the set of voxels to be updated and the camera center position is less than or equal to a first preset distance; based on the semantic point cloud information in the three-dimensional semantic information, each voxel to be updated is updated to obtain each updated voxel; and the voxels to be updated in the initial three-dimensional voxel map are replaced with corresponding updated voxels to obtain an updated three-dimensional voxel map.

[0159] In some embodiments, the first updating module 4553 is also used to obtain, for each voxel to be updated, the first truncated signed distance, the first weight and the first semantic category probability corresponding to the voxel to be updated; obtain the updated truncated signed distance and the second weight corresponding to the voxel to be updated from the semantic point cloud information; use the first weight and the second weight to fuse the first truncated signed distance and the updated truncated signed distance to obtain the second truncated signed distance; obtain the historical semantic category of the voxel to be updated, perform maximum likelihood estimation on the historical semantic category, and obtain the second semantic category probability; replace the first truncated signed distance corresponding to the voxel to be updated with the second truncated signed distance, and replace the first semantic category probability with the second semantic category probability to obtain the updated voxel.

[0160] In some embodiments, the semantic instance includes a first object instance and a first text instance, and the second update module 4554 is further used to extract mesh data from the updated three-dimensional voxel map, wherein the mesh data includes at least multiple vertices and a semantic category of each vertex; based on a second distance between each vertex and the camera center position, at least one first target vertex is determined from the multiple vertices, and the second distance between the first target vertex and the camera center position is less than or equal to a second preset distance; clustering is performed on at least one first target vertex to obtain a first clustering cluster, and the first clustering cluster is determined as the first object instance; at least one second target vertex whose semantic category is a text category is determined from the multiple vertices, and at least one second target vertex is clustered to obtain a second clustering cluster; and the second clustering cluster is determined as the first text instance.

[0161] In some embodiments, the second update module 4554 is also used to determine a third distance between different first target vertices, and based on the third distance, cluster at least one first target vertex to obtain a first cluster cluster; obtain the three-dimensional coordinates of the text content in the camera coordinate system from the three-dimensional text mark array in the three-dimensional semantic information; determine the fourth distance between each second target vertex and the three-dimensional coordinate, and based on the fourth distance, cluster at least one second target vertex to obtain a second cluster cluster.

[0162] In some embodiments, the semantic instance includes a first object instance and a first text instance, and the second update module 4554 is further used to determine the overlap rate between the first object instance and the second object instance when the initial semantic map includes a second object instance that meets a first proximity condition with the first object instance; when the overlap rate is greater than a first preset threshold, merge the first object instance and the second object instance to obtain a merged object instance; when the initial semantic map includes a second text instance that meets a second proximity condition with the first text instance, determine the ratio of identical characters between the first text instance and the second text instance; when the ratio of identical characters is greater than a second preset threshold, merge the first text instance and the second text instance to obtain a merged text instance; replace the second object instance in the initial semantic map with the merged object instance, and replace the second text instance with the merged text instance to obtain an updated semantic map.

[0163] In some embodiments, the second update module 4554 is also used to determine a candidate object instance from the initial semantic map, where the candidate object instance has the same semantic category as the first object instance; determine a fifth distance between the candidate object instance and the first object instance; and when the fifth distance is less than a third preset threshold, determine the candidate object instance as a second object instance that satisfies a first proximity condition with the first object instance.

[0164] In some embodiments, the second update module 4554 is also used to determine a candidate text instance from the initial semantic map and determine a sixth distance between the candidate text instance and the first text instance; when the sixth distance is less than a fourth preset threshold, the candidate text instance is determined as a second text instance that satisfies a second proximity condition with the first text instance.

[0165] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions or computer programs from the computer readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the electronic device executes the map data processing method described in the embodiment of the present application.

[0166] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the map data processing method provided by the embodiment of the present application, for example, Figure 3A The map data processing method shown.

[0167] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0168] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0169] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0170] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0171] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A map data processing method, characterized in that: The method comprises: Acquire a scene image and a depth image of a target scene, and perform two-dimensional information detection on the scene image to obtain object information and text information; Perform pixel coordinate association processing on the object information, the text information and the depth image to obtain three-dimensional semantic information; Determining an initial three-dimensional voxel map and an initial semantic map, and updating the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map; Semantic instances are extracted from the updated three-dimensional voxel map, and the initial semantic map is updated based on the semantic instances to obtain an updated semantic map.

2. The method according to claim 1, characterized in that The object information includes an object category and a semantic image mask, the text information includes text content and text position information, and the object information, the text information, and the depth image are subjected to pixel coordinate association processing to obtain three-dimensional semantic information, including: Performing point cloud computing processing using camera internal parameters and the depth image to obtain a three-dimensional point cloud; Based on the semantic image mask, obtaining a first object point cloud from the three-dimensional point cloud, performing color mapping processing on the object category, and obtaining color information corresponding to the object category; Assigning color information corresponding to the object category to the first object point cloud to obtain a second object point cloud; Based on the text position information, obtaining a first text point cloud from the three-dimensional point cloud, and obtaining a preset color value corresponding to the text category; Assigning the preset color value to the first text point cloud to obtain a second text point cloud; Three-dimensional semantic information is generated based on the second object point cloud, the second text point cloud and the text content.

3. The method according to claim 2, characterized in that The three-dimensional semantic information includes semantic point cloud information and a three-dimensional text mark array, and the generating of the three-dimensional semantic information based on the second object point cloud, the second text point cloud and the text content includes: Combining the second object point cloud and the second text point cloud to obtain semantic point cloud information; Acquire the three-dimensional text coordinates of the text content in the camera coordinate system from the second text point cloud; The three-dimensional text coordinates and the text content are combined to obtain a three-dimensional text mark array.

4. The method according to claim 1, characterized in that The updating of the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map includes: Determine a camera center position, and based on the camera center position, determine a voxel set to be updated from the initial three-dimensional voxel map, wherein a first distance between each voxel to be updated in the voxel set to be updated and the camera center position is less than or equal to a first preset distance; Based on the semantic point cloud information in the three-dimensional semantic information, each of the voxels to be updated is updated to obtain each updated voxel; The voxels to be updated in the initial three-dimensional voxel map are replaced with the corresponding updated voxels to obtain an updated three-dimensional voxel map.

5. The method according to claim 4, characterized in that The updating of each voxel to be updated based on the semantic point cloud information in the three-dimensional semantic information to obtain each updated voxel includes: For each voxel to be updated, obtaining a first truncated signed distance, a first weight, and a first semantic category probability corresponding to the voxel to be updated; Acquire the updated truncated signed distance and the second weight corresponding to the voxel to be updated from the semantic point cloud information; Using the first weight and the second weight, the first truncated signed distance and the updated truncated signed distance are merged to obtain a second truncated signed distance; Acquire the historical semantic category of the voxel to be updated, perform maximum likelihood estimation on the historical semantic category, and obtain a second semantic category probability; The first truncated signed distance corresponding to the voxel to be updated is replaced by the second truncated signed distance, and the first semantic category probability is replaced by the second semantic category probability to obtain an updated voxel.

6. The method according to claim 4, characterized in that The semantic instance includes a first object instance and a first text instance, and extracting the semantic instance from the updated three-dimensional voxel map includes: Extracting mesh data from the updated three-dimensional voxel map, wherein the mesh data at least includes a plurality of vertices and a semantic category of each vertex; Determine at least one first target vertex from the plurality of vertices based on a second distance between each of the vertices and the camera center position, wherein the second distance between the first target vertex and the camera center position is less than or equal to a second preset distance; Performing clustering processing on the at least one first target vertex to obtain a first cluster, and determining the first cluster as the first object instance; Determine at least one second target vertex whose semantic category is a text category from the multiple vertices, and perform clustering processing on the at least one second target vertex to obtain a second cluster cluster; The second cluster is determined as the first text instance.

7. The method according to claim 6, characterized in that The clustering of the at least one first target vertex to obtain a first cluster includes: Determine a third distance between different first target vertices, and based on the third distance, perform clustering processing on at least one first target vertex to obtain a first cluster cluster; The clustering of the at least one second target vertex to obtain a second cluster includes: Acquire the three-dimensional coordinates of the text content in the camera coordinate system from the three-dimensional text mark array in the three-dimensional semantic information; A fourth distance between each of the second target vertices and the three-dimensional coordinates is determined, and based on the fourth distance, clustering processing is performed on the at least one second target vertex to obtain a second cluster cluster.

8. The method according to any one of claims 1 to 7, characterized in that: The semantic instance includes a first object instance and a first text instance, and the updating of the initial semantic map based on the semantic instance to obtain an updated semantic map includes: When the initial semantic map includes a second object instance that satisfies a first proximity condition with the first object instance, determining an overlap rate between the first object instance and the second object instance; When the overlap rate is greater than a first preset threshold, merging the first object instance and the second object instance to obtain a merged object instance; When the initial semantic map includes a second text instance that satisfies a second proximity condition with the first text instance, determining a ratio of identical characters between the first text instance and the second text instance; When the same character ratio is greater than a second preset threshold, merging the first text instance with the second text instance to obtain a merged text instance; The second object instance in the initial semantic map is replaced by the merged object instance, and the second text instance is replaced by the merged text instance to obtain an updated semantic map.

9. The method according to claim 8, characterized in that The method further comprises: Determining a candidate object instance from the initial semantic map, the candidate object instance having the same semantic category as the first object instance; determining a fifth distance between the candidate object instance and the first object instance; When the fifth distance is less than a third preset threshold, the candidate object instance is determined to be a second object instance that satisfies a first proximity condition with the first object instance.

10. The method according to claim 8, characterized in that The method further comprises: determining a candidate text instance from the initial semantic map, and determining a sixth distance between the candidate text instance and the first text instance; When the sixth distance is less than a fourth preset threshold, the candidate text instance is determined to be a second text instance that satisfies a second proximity condition with the first text instance.

11. A map data processing device, characterized in that: The device comprises: A first acquisition module is used to acquire a scene image and a depth image of a target scene, and perform two-dimensional information detection on the scene image to obtain object information and text information; A second acquisition module is used to perform pixel coordinate association processing on the object information, the text information and the depth image to obtain three-dimensional semantic information; A first updating module, configured to determine an initial three-dimensional voxel map and an initial semantic map, and update the initial three-dimensional voxel map based on the three-dimensional semantic information to obtain an updated three-dimensional voxel map; The second updating module is used to extract semantic instances from the updated three-dimensional voxel map, and update the initial semantic map based on the semantic instances to obtain an updated semantic map.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the map data processing method according to any one of claims 1 to 10 when executing the computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the map data processing method according to any one of claims 1 to 10 is implemented.

Images