Semantic map data updating method and related devices

By processing image sequences and depth images, and combining similarity thresholds with point cloud data fusion, the accuracy problem of semantic map data updates was solved, enabling object-level updates and accurate representations, thus improving the real-time performance and efficiency of semantic maps.

CN123068435APending Publication Date: 2026-11-10HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510601209.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

How to improve the accuracy of semantic map data updates, especially when the position and shape of objects in the environment change, and avoid repeatedly building semantic map data.

Method used

By acquiring image sequences and depth images, object features and point cloud data are determined. Semantic map data is updated using similarity thresholds. By combining point cloud data fusion and object disappearance judgment, the semantic map is accurately updated.

Benefits of technology

It achieves object-level semantic map data updates, improves the accuracy and processing efficiency of semantic map data, reduces redundant calculations, and ensures the real-time performance and accuracy of map data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068435A_ABST
    Figure CN123068435A_ABST
Patent Text Reader

Abstract

Embodiments of the present application apply to the technical field of terminals, and provide a semantic map data updating method and related equipment, which comprises: acquiring a first image sequence and a depth image corresponding to each frame of the first image sequence, any frame of the first image sequence and the depth image corresponding to the first image are obtained by collecting the same scene; determining a first object feature of a first object recorded in the first image sequence according to the first image sequence; determining first point cloud data of the first object according to the first image sequence and the depth image corresponding to each frame of the first image; and updating the semantic map data according to the similarity between a second object feature of a second object in the semantic map data and the first object feature, and the first point cloud data, the similarity between the object features of different objects being less than a similarity threshold. Based on the technical method of the present application, the accuracy of semantic map data updating can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a semantic map data update method and related equipment. Background Technology

[0002] Semantic maps, also known as semantic graphs, are a type of map that combines physical spatial information with semantic information. They not only describe the layout and structure of physical space but also include semantic interpretations of objects, scenes, and spatial relationships within the environment. For example, they not only label the location and shape of objects but also assign semantic labels to these objects and spaces. Semantic maps are widely used in fields such as autonomous driving and intelligent robotics.

[0003] A semantic map is a map that contains semantic information, and semantic map data can be understood as the collection of information used to construct a semantic map. The positions or shapes of objects in the environment represented by semantic map data may change. To ensure that semantic map data accurately represents the environment, it can be updated. Improving the accuracy of semantic map updates is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a semantic map data update method and related equipment, which can improve the accuracy of the updated semantic map.

[0005] In a first aspect, a semantic map data update method is provided. The method includes: acquiring a first image sequence and a depth image corresponding to each frame of the first image sequence, wherein any frame of the first image sequence and the depth image corresponding to the first frame of the first image sequence are obtained by collecting data from the same scene; determining a first object feature of a first object recorded in the first image sequence based on the first image sequence; determining first point cloud data of the first object based on the first image sequence and the depth image corresponding to each frame of the first image; and updating the semantic map data based on the similarity between a second object feature of a second object and the first object feature in the semantic map data, and the first point cloud data, wherein the similarity between the object features of different objects is less than a similarity threshold.

[0006] The semantic map data update method provided in this application determines whether the first object recorded in the first image sequence and the second object recorded in the semantic map data are the same object because the similarity between the object features of different objects is less than a similarity threshold. Since the similarity between the object features of different objects is less than the similarity threshold, the semantic map data includes the object features of the second object, thus achieving an accurate representation of objects in the environment. The semantic map data is updated based on whether the same object is recorded, allowing for updates to existing semantic map data over time based on newly acquired image sequences. This eliminates the need to repeatedly create semantic map data for the same location, and the update is object-level, improving the accuracy of the semantic map data.

[0007] In some possible embodiments, determining the first object feature of the first object recorded in the first image sequence based on the first image sequence includes: determining the first object feature and the object contour of the first object included in the first object image in the first image sequence based on the first image sequence, wherein the first object image is a first image recording the first object; determining the first point cloud data of the first object based on the first image sequence and the depth image corresponding to each frame of the first image includes: determining the first point cloud data based on the depth image corresponding to each frame of the first object image and the contour of the first object in each frame of the first object image.

[0008] The contour of the first object in the first object image is used during the determination of the first point cloud data. The first object features can be obtained during the process of determining the contour of the first object in the first object image. Therefore, the first object features do not need to be obtained through additional operations, improving the processing efficiency of the semantic map data update method.

[0009] In some possible implementations, determining the first object feature of the first object recorded in the first image sequence based on the first image sequence includes: determining the object outline of the first object included in the first object image in the first image sequence, wherein the first object image is a first image recording the first object; and determining the first object feature based on the object outline of the first object and the first object image in which the first object is located.

[0010] Based on the first object image recorded in the first image sequence, and the outline of the first object in the first object image, the features of the first object are determined, thereby improving the accuracy of the features of the first object.

[0011] In some possible implementations, determining the object contour of the first object included in the first object image of the first image sequence includes: performing object recognition on each frame of the first image in the first image sequence to determine the position information of the first object in the first object image; and determining the object contour of the first object by performing object segmentation processing on the first image sequence based on the position information of the first object in the first object image.

[0012] The position information of the first object in the first object image can represent the position of the recognition box where the first object is located in the first object image. By performing object recognition on each frame of the first image, the position information of the first object in each frame of the first object image is determined, and object segmentation processing is performed on the first image sequence based on the position information of the first object in each frame of the first object image. This can determine the object contour of the first object in each frame of the first object image, thereby improving the accuracy of the object contour of the first object.

[0013] In some possible implementations, updating the semantic map data based on the similarity between the second object features of the second object and the first object features in the semantic map data, and the first point cloud data, includes: if the target second object is not included in the second objects included in the semantic map data, adding the first object features and the first point cloud data to the semantic map data, wherein the similarity between the second object features of the target second object and the first object features is greater than or equal to the similarity threshold; if the target second object is included in the second objects included in the semantic map data, determining the second point cloud data of the target second object in the semantic map data based on the first point cloud data.

[0014] If the similarity between the second object features of the target second object and the first object features is greater than or equal to the similarity threshold, then the target second object can be understood as an object recorded in the semantic map data that is the same as the first object. If the semantic map data includes the target second object, the second point cloud data of the target second object is determined based on the first point cloud data of the first object. If the semantic map data does not include the target second object, the first object is treated as a new second object, and the first object features and first point cloud data of the first object are added to the semantic map data, making the updated semantic map data more accurate.

[0015] In some possible implementations, when the target second object is included among the second objects included in the semantic map data, updating the second point cloud data of the target second object in the semantic map data based on the first point cloud data includes: when the target second object is included in the semantic map data and the target second object meets an overlap condition, fusing the second point cloud data of the target second object with the first point cloud data to obtain new second point cloud data of the target second object, wherein the overlap condition is: the semantic map data includes the second point cloud data of the target second object, and the overlap degree between the second point cloud data of the target second object and the first point cloud data is greater than or equal to an overlap degree threshold; when the target second object is included in the semantic map data and the target second object does not meet the overlap condition, using the first point cloud data as the second point cloud data of the target second object.

[0016] When the target second object meets the overlap condition, the second point cloud data of the target second object in the semantic map data has a high degree of overlap with the first point cloud data of the first object. By fusing the second point cloud data with the first point cloud data, a new second point cloud data of the target second object is obtained, which makes up for the deficiencies of single data and makes the new point cloud data more accurately and completely reflect the shape of the target second object.

[0017] The second target object does not meet the overlap condition. It can be that the second point cloud data of the second target object is not included in the semantic map data, or it can be that the second point cloud data of the second target object is included in the semantic map data, and the overlap between the second point cloud data of the second target object and the first point cloud data of the first object in the semantic map data is low.

[0018] If the second point cloud data of the second object is not recorded in the semantic map, the first point cloud data of the first object can be used as the second point cloud data of the second object.

[0019] When the overlap between the second point cloud data of the target second object and the first point cloud data of the first object in the semantic map is low, the position of the target second object in the environment has changed. Neither the first nor the second point cloud data may fully reflect the object's shape in all orientations, leading to high difficulty and low accuracy in point cloud registration. Therefore, when the overlap between the second point cloud data of the target second object and the first point cloud data of the first object is low, replacing the second point cloud data of the target second object with the first point cloud data of the first object improves the accuracy of the point cloud data.

[0020] In some possible implementations, the method further includes: determining candidate second objects among the second objects included in the semantic map data, wherein the candidate second objects are located within the frustum range corresponding to at least one frame of the first image in the first image sequence and are not occluded; and if the candidate second objects include disappearing objects, deleting the second point cloud data of the disappearing objects from the semantic map data, wherein the similarity between the second object features of the disappearing objects and the first object features is less than or equal to the similarity threshold.

[0021] For a candidate second object that is located within the view frustum corresponding to at least one first image and is not occluded, the second object features of the candidate second object are compared with the first object features of the first object to determine whether the candidate second object has disappeared in the environment represented by the first image sequence compared to the environment represented by the semantic map data. If the candidate second object has disappeared, the second point cloud data of the candidate second object is deleted to improve the accuracy of the updated semantic map data.

[0022] In some possible implementations, the semantic map data further includes the category of each second object. Updating the semantic map data based on the similarity between the second object features of the second objects in the semantic map data and the features of the first object, and the first point cloud data, includes: determining second objects of the same category as the first object based on the categories of the second objects in the semantic map data; calculating the similarity between the second object features of the second objects of the same category and the features of the first object; and updating the semantic map data based on the similarity between the second object features of the second objects of the same category and the features of the first object, and the first point cloud data.

[0023] Objects belonging to the same category are considered identical. Similarity is calculated for objects within the same category to determine if they are truly the same. However, similarity calculations are not performed for objects in different categories, improving processing efficiency and reducing the computational burden of updating semantic map data.

[0024] Secondly, a semantic map updating apparatus is provided, including a unit for performing the method of the first aspect. This apparatus may be a terminal device or a chip within the terminal device.

[0025] Thirdly, an electronic device is provided, including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, causing the electronic device to perform the method of the first aspect.

[0026] Fourthly, a chip is provided, including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to implement the method of the first aspect.

[0027] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer program code for implementing the method of the first aspect.

[0028] In a sixth aspect, a computer program product is provided, the computer program product comprising: computer program code, the computer program code being used to implement the method of the first aspect. Attached Figure Description

[0029] Figure 1 It is a schematic diagram of a semantic map and the images used in the process of creating a semantic map;

[0030] Figure 2 This is a schematic structural diagram of a semantic map data updating device provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of object recognition in an image;

[0032] Figure 4 This is a schematic diagram of the result of object segmentation;

[0033] Figure 5 This is a schematic structural diagram of an object segmentation model provided in an embodiment of this application;

[0034] Figure 6 This is a schematic structural diagram of another semantic map data updating device provided in the embodiments of this application;

[0035] Figure 7 This is a schematic flowchart of a semantic map update method based on object features provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of a semantic map represented by updated semantic map data provided in an embodiment of this application;

[0037] Figure 9 This is a schematic flowchart illustrating a system operation method provided in an embodiment of this application;

[0038] Figure 10 This is a schematic flowchart illustrating a semantic map data update method provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of a hardware system for an electronic device applicable to this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] It should be understood that "at least one" in this application refers to one or more items, and "more than one" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0042] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0043] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties (e.g., the user has given explicit consent, the user has been notified, etc.), and the collection, use and processing of the relevant data must comply with the relevant regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0044] To facilitate understanding, before introducing the image processing method provided in the embodiments of this application, the terms or nouns involved in the embodiments of this application will be briefly introduced.

[0045] (1) Artificial Intelligence

[0046] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0047] (2) Machine Learning

[0048] Machine learning is an important branch of artificial intelligence, and deep learning is an important branch of machine learning. Deep learning refers to using multi-layer neural network structures to learn representations of various things in the real world that can be directly used for computer calculations (such as objects in images, sounds in audio, etc.) from big data.

[0049] (3) Neural Network

[0050] A neural network model is a mathematical method for simulating the actual neural network in humans. A neural network model is a complex network system formed by extensively interconnecting a large number of simple processing units (called neurons or neural units). A neural network model is a network formed by linking many of these individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0051] (4) Deep Neural Networks

[0052] Deep neural networks (DNNs), also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of the layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix (a weight matrix formed by vectors from many layers) of the trained deep neural network.

[0053] (5) Loss Function

[0054] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0055] (6) Backpropagation algorithm

[0056] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0057] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0058] A semantic map, also known as a semantic graph, is a type of map that combines physical spatial information with semantic information. A semantic map not only describes the layout and structure of physical space but also includes semantic interpretations of objects, scenes, and spatial relationships within the environment. For example, it not only labels the location and shape of objects but also assigns semantic labels to these objects and spaces. Semantic maps are widely used in fields such as autonomous driving and intelligent robotics.

[0059] A semantic map is a map that contains semantic information, and semantic map data can be understood as the collection of information used to construct a semantic map. Semantic map data is the "raw material" for a semantic map; by processing and organizing it, a semantic map can be constructed.

[0060] Semantic map data can include point cloud data and categories for each object across multiple objects. A semantic map can be a map with spatial and semantic relationships formed by organizing semantic map data according to certain rules and logic. For example... Figure 1 The semantic map shown in (a) can record objects that may include items 101 to 106 in the room. Among them, item 101 is a door, item 102 is a wall, item 103 is a bed, item 104 is a cabinet, item 105 is the floor, item 106 is a chair, etc. Figure 1 The semantic map shown in (a) can be built based on image sequences and depth image sequences. Figure 1 (b) in the text can be used to establish Figure 1 The image in (a) is a frame from the image sequence used in the process of creating the semantic map.

[0061] Point cloud data for each object can be understood as a set of points used to describe the surface shape and features of that object. Each point in the point cloud data can contain three-dimensional coordinates and can also carry color information or other attributes. Color information can be represented using the red-green-blue (RGB) color space, the hue-saturation-value (HSV) color space, or the International Commission on Illumination (CIE) XYZ color space, among other methods.

[0062] The sensor used to collect environmental data can be a color depth (RGB-depth, RGBD) camera or other sensors. An RGB-depth (RGBD) camera is a sensor capable of simultaneously acquiring a color image of a scene and depth information (D). Depth information represents the distance to objects in the scene. Figure 1 The image shown in (b) can be captured by an RGBD camera.

[0063] RGBD cameras capture color information of a scene using an image sensor. RGBD cameras acquire depth information through one or more methods, including infrared structured light, time-of-flight (ToF) imaging, and binocular stereo vision. Infrared structured light projects infrared light with a specific structure (such as a dot matrix or grating) onto an object and uses an infrared camera to capture the reflection pattern of this light on the object's surface, thus calculating the object's depth. Time-of-flight imaging measures the time it takes for light to travel from the camera to the object and back, using this time difference and the speed of light to calculate the object's distance. RGBD cameras combine traditional RGB imaging technology with depth information capture capabilities, enabling real-time reconstruction of a color stereoscopic data space and providing rich 3D scene information.

[0064] The form and location of objects in the environment may change dynamically. If the semantic map data is rebuilt every time new environmental data is acquired, the computational load is substantial. To ensure that the semantic map data accurately represents the environment, it can be updated. How to accurately update objects in the semantic map is a problem that urgently needs to be solved.

[0065] To address the aforementioned issues, this embodiment provides a semantic map data updating device and a semantic map data updating method.

[0066] Figure 2This is a schematic structural diagram of a semantic map data update device provided in an embodiment of this application.

[0067] The semantic map data update device 200 is used to process the first image sequence, the depth image sequence, and the camera parameters to update the semantic map data.

[0068] The first image sequence can represent a continuous video. That is, each first image in the first image sequence can be understood as a frame in the video. The depth image sequence can include the depth image corresponding to each first image in the first image sequence. Any first image and its corresponding depth image in the first image sequence are obtained by capturing the same scene. For example, an electronic device including the semantic map data update device 200 can capture the first image and its corresponding depth image at the same time and in the same direction. During the data acquisition process to obtain the first image sequence and the depth image sequence, the electronic device can be stationary or moving. For example, the electronic device can be a robot or other mobile electronic device. During the movement of the electronic device, the first image sequence can be captured by a camera, and the depth image sequence can be captured by a distance sensor. For example, the electronic device can include an RGBD camera. The RGBD camera can include a camera and a distance sensor.

[0069] The semantic map data update device 200 includes an object recognition model 210, a segmentation and tracking model 220, a point cloud generation module 230, an update module 240, and an object feature extraction model 250.

[0070] Object recognition model 210 is used to perform object recognition on each first image in the first image sequence, obtaining the position information of the recognition box of the first object in each first image. The position information of the recognition box is used to indicate the position of the recognition box in the first image. Different recognition boxes corresponding to the first image correspond to different objects. The number of first objects in each first image can be one or more, and thus the number of recognition boxes corresponding to each first image can be one or more.

[0071] Object recognition is an important task in computer vision, also known as target identification or target detection. Object recognition allows us to detect recorded objects in an image, that is, to determine the position and size of recorded objects. The position and size of an object can be represented by the position information of a bounding box. Recorded objects in an image can be located within the bounding box.

[0072] Object recognition can be achieved through methods such as feature matching or deep learning. In feature matching, key features (such as corners, edges, and textures) are extracted from an image, and then these features are matched to identify objects. In deep learning, an initial object recognition model is trained using recognition training data to obtain a new object recognition model.

[0073] Object recognition model 210 performs object recognition on the first image to obtain the position information of the recognition box of the first object in the first image. The first object is recorded in the recognition box of a certain first object in the first image. Object recognition model 210 can be, for example, a recognize anything model (RAM).

[0074] Object recognition can also classify the recorded objects in an image, that is, determine the category of the objects in the image. In other words, the object recognition model 210 can also determine the category of the first object in each recognition box by processing each first image.

[0075] Figure 3 Image (a) shows a first image in the first image sequence. Object recognition model 210 pairs... Figure 3 Object recognition is performed on the first image shown in (a) to obtain the position information of each of the multiple recognition boxes 321 to 326. These multiple recognition boxes... Figure 3 The position in the first image shown in (a) is as follows: Figure 3 As shown in (b) of the image. In the first image, the category of item 311 in the recognition box 321 is door, the category of item 313 in the recognition box 322 is wall, the category of item 313 in the recognition box 323 is bed, the category of item 314 in the recognition box 324 and the category of item 316 in the recognition box 326 are chair, and the category of item 315 in the recognition box 325 is ground.

[0076] The segmentation and tracking module 220 processes the position information of the bounding boxes of the first image sequence and the first object to obtain the contour of each first object in at least one first object image. Through the processing of the segmentation and tracking module 220, it can be determined whether the same object is recorded in any two first images of the first image sequence.

[0077] The segmentation tracking model 220 may include a model for segmenting objects in an image, such as the Segment Anything model (SAM). Alternatively, the segmentation tracking model 220 may be a model for video object segmentation (VOS), such as SAM2. Or, the segmentation tracking model 220 may segment and track objects recorded in an image sequence based on feature matching, optical flow, or other methods.

[0078] The segmentation and tracking module 220 can output an object mask sequence corresponding to each first object. The object mask sequence corresponding to each first object can include the object mask of the first object in at least one first image of the first image sequence. The object mask of the first object in a certain first image can be understood as the contour of the first object in that first image. Alternatively, each object mask corresponding to a first image can be used to represent the contour of the first object corresponding to that object mask in the first image.

[0079] In the first image sequence, a certain first object may be recorded in a portion of the first images, rather than in all the first images. The image in the first image sequence that records the first object can be referred to as the first object image of the first object. That is, the object mask sequence of the first object may include the object mask corresponding to the first object image that records the first object.

[0080] The segmentation tracking module 220 can also be used to process the first image sequence, the position information of the recognition box corresponding to each first image in the first image sequence, and the category corresponding to each recognition box to obtain the object mask sequence corresponding to each first object. The category of the first object can be represented by text. Using the category of the first object as the basis for the segmentation tracking module 220 to segment the first object recorded in each first object image in the first image sequence improves the accuracy of the determined contour of the first object.

[0081] Figure 4 The segmentation tracking model 220 pairs includes Figure 3 The first image sequence of the first image shown in (a) and the recognition box corresponding to each frame of the first image are processed to obtain the following result. Figure 3 The object mask corresponding to the first image shown in (a) is shown. For Figure 3 The recognition bounding box obtained from the first image recognition shown in (a) is shown in the figure. Figure 4 (a) to (f) in the figure can be the object mask of the first object recorded in the recognition boxes 321 to 326, respectively.

[0082] The image segmentation model in segmentation tracking model 220 can be a SAM model. The SAM model architecture includes an image encoder, a prompt encoder, and a mask decoder, used to segment objects in an image. When processing an image, the image encoder encodes the input image to obtain image features. Image features can also be called image embeddings. The prompt encoder extracts features from one or more prompts, such as text, bounding boxes, masks, and prompt points, to obtain prompt features. Bounding boxes in the prompt information can also be understood as the positional information of the bounding boxes. Prompt features can also be called prompt embeddings. The SAM model can fuse image features and prompt features. The mask decoder decodes the fused result of image features and prompt features to generate a mask. The mask generated by the mask decoder represents the contour of the object indicated by the prompt information in the input image.

[0083] The segmentation and tracking model 220 can also be the SAM2 model. The SAM2 model is used to segment objects in each frame of a video. Compared to the SAM model, the SAM2 model also introduces motion-aware instance-level memory. The motion-aware instance-level memory mechanism allows the model to memorize the features of objects in the temporal dimension and combine it with motion modeling to optimize mask generation. Thus, the SAM2 model can process image sequences and can dynamically update and utilize the feature information of objects.

[0084] The architecture of the SAM2 model can be found in [reference needed]. Figure 5 The SAM2 model 500 may include an image encoder 501, a memory attention module 502, a cue encoder 503, a mask decoder 504, a memory encoder 505, and a memory bank 506. Instance-level memory for motion perception is achieved through the memory encoder 505, the memory bank 506, and the memory attention module 502.

[0085] The image sequence processed by the SAM2 model 500 includes the i-th frame image. The following describes the processing procedure of the SAM2 model 500 using the processing procedure of the i-th frame image as an example.

[0086] The image encoder 501 is used to extract features from the i-th frame image to obtain the image features of the i-th frame image.

[0087] The memory attention 502 is used to fuse the image features of the i-th frame with the memory features corresponding to the (i-1)-th frame in the memory bank to obtain the fused features corresponding to the i-th frame. The i-th frame can be any frame in the image sequence. If the i-th frame is the first frame in the image sequence, the memory attention 502 can use the image features of the first frame as the fused features corresponding to the first frame.

[0088] The prompt encoder 503 is used to extract features from the prompt information corresponding to the i-th frame image to obtain the prompt features corresponding to the i-th frame image.

[0089] The mask decoder 504 is used to generate the i-th mask based on the fused features corresponding to the i-th frame image and the cue features corresponding to the i-th frame image. The mask decoder 504 can fuse the fused features corresponding to the i-th frame image and the cue features corresponding to the i-th frame image, and then decode the fused features; the result of the decoding is the i-th mask. The mask sequence obtained by the SAM2 model 500 for processing the image sequence includes the i-th mask.

[0090] The i-th mask represents the outline of an object in the i-th frame image. In some cases, all or part of the images in the image sequence do not have corresponding prompt information. In the case that the i-th frame image does not have corresponding prompt information, the mask decoder 504 can decode the fusion features corresponding to the i-th frame image to obtain the i-th mask.

[0091] The memory encoder 505 is used to extract features from the i-th mask to obtain the mask features corresponding to the i-th frame image.

[0092] The memory bank 506 is used to fuse the mask features corresponding to the i-th frame image and the image features of the i-th frame image to obtain the memory features corresponding to the i-th frame image.

[0093] During the process of SAM2 model 500 as segmentation and tracking model 220 processing the first image sequence and the bounding box of the first object in each first image, the bounding box of the first object in each first image can serve as the prompt information corresponding to that first image.

[0094] After the segmentation and tracking model 220 processes the first image sequence to obtain the object mask sequence of the first object, the semantic map data update device 200 can process multiple first object images of the first object, as well as the contour of the first object in each first object image and / or the position information of the recognition box of the first object in each first object image, through the object feature extraction model 250, to obtain the first object features of the first object. The contour of the first object in each first object image can be represented by the object mask sequence of the first object.

[0095] Object feature extraction is a core task in machine learning, computer vision, and natural language processing, aiming to extract representative and discriminative features from raw data. The first object's features can include one or more of the following: color features, texture features, shape features, spatial features, and semantic features. Color features can be represented by histograms of pixel colors or color moments. Texture features can be represented by gray-level co-occurrence matrices, which describe the joint distribution of gray values ​​among pixels. Spatial features can be represented by one or more of the following: scale-invariant feature transform (SIFT) features, speed-up robust features (SURF) features, etc.

[0096] The feature extraction of the first object in each first image can be performed by extracting features from the images within each bounding box in the first image, or by extracting features from the images within the contours represented by each mask in the first image. The object feature extraction model 250 can be used to extract features of the first object in each first object image and fuse the extracted features to obtain the first object features of the first object. Alternatively, the object feature extraction model 250 can extract features from multiple first object images based on the position information of the bounding box of the first object in each first object image and / or the contour of the first object in each first object image to obtain the first object features of the first object.

[0097] For example, the object feature extraction model 250 can be a neural network model such as data intensive object detection with self-supervision v2 (DINOv2) or SAM2. The first object feature of the first object can be a high-level feature output by the object feature extraction model 250.

[0098] DINOv2 employs self-supervised learning to learn rich feature representations from large amounts of unlabeled data. This learning approach allows the DINOv2 model to not only focus on object category information but also capture detailed features such as shape, texture, and color. Therefore, even objects belonging to the same category will exhibit differences in these detailed features, leading to variations in the features extracted by the DINOv2 model.

[0099] In other embodiments, such as Figure 6As shown, the semantic map data update device 200 may not include the object feature extraction model 250. During the process of processing the first image sequence to obtain the object mask sequence of the first object, the segmentation tracking model 220 can also determine the first object features of the first object.

[0100] It should be understood that when the segmentation tracking model 220 determines the object mask sequence of each first object in the first image sequence based on optical flow, the semantic map data update device 200 may include the object feature extraction model 250. However, when the segmentation tracking model 220 segments objects in the image based on object features, the semantic map data update device 200 may not include the object feature extraction model 250.

[0101] Taking the segmentation and tracking model 220 as the SAM2 model as an example, the features of the object are the basis for the SAM2 model to generate the mask. During the process of the SAM2 model processing the first image sequence to obtain the object mask sequence of the first object, the first object features of the first object can be extracted during the process of the mask decoder generating the object mask of the first object.

[0102] The segmentation and tracking model 220, used to segment objects in an image, can be called an image object segmentation model. The image object segmentation model can process each first object image and the position information of the first object's bounding box in that first object image separately, obtaining the mask of the first object in the first object image and the features of the first object in that first object image. The mask of the first object in the first object image represents the outline of the first object in the first object image. When there are multiple first object images recording the first object, the segmentation and tracking model 220 can fuse the features of the first object in each of the first object images to obtain the first object features.

[0103] During the process of object recognition model 210 recognizing each first image in the first image sequence, features of the first object can be extracted. Similarly, during the process of segmentation and tracking model 220 segmenting the first image sequence, features of the first object can also be extracted. However, the features extracted by object recognition model 210 are primarily used to determine the category and bounding box of the first object. For objects of the same category but different appearances, the similarity of the features extracted by object recognition model 210 is high; that is, the features extracted by object recognition model 210 cannot be used to distinguish between different objects. In contrast, the features extracted by segmentation and tracking model 220 are primarily used to distinguish between foreground and background, i.e., to accurately determine the outline of the first object. The accuracy of the features extracted by segmentation and tracking model 220 in distinguishing between different objects remains relatively low.

[0104] The object feature extraction model 250 extracts features from different objects that differ significantly, exceeding a preset threshold. The first object feature extracted by the object feature extraction model 250 is used for subsequent updates to the semantic map data. By setting the object feature extraction model 250 in the semantic map data update device 200 to extract the first object feature, the updated semantic map data becomes more accurate.

[0105] After the segmentation and tracking model 220 processes and obtains the contour of the first object in each first object image, the first point cloud data of the first object can be generated by the point cloud generation module 230.

[0106] The point cloud generation module 230 is used to generate first point cloud data for each first object based on the depth image sequence, the object mask sequence corresponding to each first object, and camera parameters. The first point cloud data for each first object can be understood as a set of discrete points in three-dimensional space. Each point contains spatial coordinates, which are the coordinates of that point in three-dimensional space.

[0107] For each first object, based on an object mask in the object mask sequence of that first object and the corresponding depth image, a single-frame point cloud of the first object in a coordinate system with the electronic device as the origin is determined. The depth image corresponding to the object mask can be understood as a depth image acquired at the same time as the first image corresponding to the object mask. The process of determining the single-frame point cloud of the first object in a coordinate system with the electronic device as the origin can also be understood as a projection of the first object from two dimensions to three dimensions.

[0108] Camera parameters can include the camera's position and orientation. An accelerometer in the electronic device can be used to determine the camera's orientation. The camera's orientation can be determined based on the electronic device's attitude. In the electronic device, the relative positions of the camera performing image acquisition and the distance sensor are fixed. Generally, the position and orientation of the camera performing image acquisition and the distance sensor are the same. Based on the camera's position and orientation, the mapping relationship between the coordinate system with the camera as the origin and a preset coordinate system can be determined, thereby performing coordinate system transformation on the single-frame point cloud data. Based on the camera parameters, the coordinate system transformation can be performed on the single-frame point cloud of the first object in the coordinate system with the electronic device as the origin, thereby obtaining the single-frame point cloud of the first object in the preset coordinate system.

[0109] Based on each object mask in the object mask sequence of the first object, multiple single-frame point clouds of the first object in a preset coordinate system are obtained, and the first point cloud data of the first object can be determined. For example, after determining the multiple single-frame point clouds of the first object in the preset coordinate system, the multiple single-frame point clouds of the first object can be fused to obtain the first point cloud data of the first object.

[0110] Each point in the first point cloud data of each first object may also include other information, such as color. If each point in the first point cloud data includes color, the point cloud generation module 230 can determine the first point cloud data of each first object based on the depth image sequence, the object mask sequence corresponding to each first object, the first image sequence, and camera parameters.

[0111] The update module 240 can update the semantic map data based on the first point cloud data and the first object features of each first object. The semantic map data includes the second object features of the second objects. The update module 240 can perform... Figure 7 The method for updating semantic map data based on object features is shown.

[0112] The following is combined Figure 7 The semantic map data update method based on object features performed by the update module 240 is described.

[0113] Figure 7 This is a schematic flowchart of a semantic map data update method based on object features provided in an embodiment of this application. Figure 7 The method 700 shown includes steps S711 to S713, and steps S721 to S724.

[0114] Semantic map data may include all or part of the second point cloud data of the second object.

[0115] Step S711: Based on the camera parameters, determine the candidate second object located within the monitoring range from the second object that includes the second point cloud data in the semantic map data.

[0116] During the acquisition of the first image sequence, the orientation and position of the camera may change.

[0117] Based on the camera parameters at the time of acquiring each first image in the first image sequence, the viewing frustum of the camera at that moment can be determined. The viewing frustum refers to the three-dimensional spatial region that the camera can capture at a given moment. The spatial region within the viewing frustum can be determined using the camera's intrinsic parameters (such as focal length) and extrinsic parameters (such as position and pose). Based on the viewing frustum at that moment, and the second point cloud data of each second object in the semantic map data, candidate second objects located within the camera's acquisition range at that moment can be identified. The second point cloud data of the candidate second objects overlaps with the viewing frustum of any first image, and the candidate second objects are not occluded by other second objects. Candidate second objects are determined based on the second point cloud data, and the semantic map data includes the second point cloud data of each candidate second object.

[0118] Candidate second objects within the monitoring range corresponding to the first image sequence may include second objects whose acquisition time in each of the first images in the first image sequence is within the acquisition range of the camera.

[0119] Step S712: Among the candidate second objects, determine whether there is a missing object, and the second object features of the missing object do not match the first object features.

[0120] The second object features of the candidate second object are compared with the first object features of each first object. If the similarity between the second object features of a candidate second object and the first object features of each first object is less than the similarity threshold, it can be determined that the second object features of the candidate second object do not match the first object features of each first object, that is, the candidate second object can be regarded as a missing object.

[0121] If there are missing objects, step S713 can be performed.

[0122] Step S713: Delete the second point cloud data of the missing object from the semantic map data.

[0123] If no object disappears, or if step S713 has been performed, the process can be terminated.

[0124] Through steps S711 to S713, it can be determined whether the second object in the semantic map data has disappeared in the first image sequence, and the semantic map data can be updated according to the disappeared object.

[0125] Steps S721 to S724 can be performed before or after steps S711 to S713, or they can be performed in parallel with steps S711 to S713.

[0126] Step S721: Determine whether the semantic map data includes a second object feature that matches the first object feature of the first object.

[0127] If it is determined that the semantic map data includes a second object feature that matches the first object feature of the first object, it can be determined that the semantic map data includes an object that is the same as the first object, and step S722 can be performed.

[0128] Step S722: Determine whether the overlap condition is met.

[0129] The overlap condition is that the semantic map data includes the second point cloud data of the target second object, and the overlap degree between the second point cloud data of the target second object and the first point cloud data of the first object is greater than or equal to the overlap degree threshold. The target second object is a second object of a second object feature object that matches the first object feature of the first object. In other words, the similarity between the first object feature of the first object and the second object feature of the target second object is greater than or equal to the similarity threshold.

[0130] If the overlap condition is met, it can be determined that the position of the first object in the first image sequence has not changed significantly from the position of the target second object, and no further processing is required.

[0131] If the overlap condition is not met, it could be that the position of the first object in the first image sequence has changed relative to the position of the target second object, or it could be that the semantic map data does not include the second point cloud data of the target second object. If the overlap condition is not met, step S723 can be performed.

[0132] Step S723: Use the first point cloud data of the first object as the second point cloud data of the second object in the semantic map data.

[0133] If the semantic map data includes the second point cloud data of the target second object, the second point cloud data of the second object in the semantic map data can be replaced with the first point cloud data of the first object to obtain new second point cloud data for the second object. If the semantic map data does not include the second point cloud data of the target second object, the first point cloud data of the first object can be added to the semantic map data as the second point cloud data of the second object.

[0134] If in step S721 it is determined that the semantic map data does not include a second object feature that matches the first object feature of the first object, it can be determined that the semantic map data does not record an object that is the same as the first object, and step S724 can be performed.

[0135] Step S724: Add the first object features and first point cloud data of the first object to the semantic map data.

[0136] If the semantic map data includes features of a second object that match the features of the first object, it can be determined that the first object and the second object that match the features are the same object, and step S722 is performed.

[0137] Through steps S721 to S724, it can be determined whether the first object is an object already recorded in the semantic map data. The point cloud data and object features of objects not recorded in the semantic map are then added to the semantic map, or the point cloud data of objects already recorded in the semantic map data are updated. By comparing the features of the detected first object with those of the second object in the semantic map data, re-identification of the second object in the semantic map data is achieved. Therefore, the semantic map data is updated based on real-time observation results.

[0138] In some embodiments, the semantic map data may further include the category of a second object. The object recognition model 210 processes the first image to obtain the category of the first object recorded in each recognition box. Steps S712 and S721 both involve determining whether the similarity between the first object features of the first object and the second object features of the second object is greater than a similarity threshold. To reduce computational load, when comparing the similarity between the first object features of a first object and the second object features of a second object recorded in the semantic map data, features of objects with the same category can be compared to determine whether the first object and the second object are the same object. For second objects recorded in the semantic map data that have a different category than the first object, since the object categories are different, it can be determined that the second object is not the same as the first object, and the similarity between object features is no longer determined, thus reducing computational load.

[0139] It should be understood that if, in step S721, it is determined that the semantic map data does not include a second object feature that matches the first object feature, the electronic device can proceed to step S724, and can also add the category of the first object to the semantic map data. If, in step S712, it is determined that a missing object exists, the category of the missing object can also be deleted from the semantic map data.

[0140] Semantic map data can also include identification (ID) and semantic tags for each second object. In other words, the object information for each second object in the semantic map data can include ID, semantic tags, category, and second point cloud data. The ID can be a string assigned to a new second object when its information is added. Semantic tags can be used to represent the corresponding category, or one or more of the following: object color, corresponding user, etc.

[0141] When a first object is added to the semantic map data as a new second object, the object information of the first object can be added to the semantic map data. If a second object is determined to be a disappearing object, its object information can be deleted from the semantic map data.

[0142] Figure 1 The semantic map shown in (a) can be understood as a semantic map represented by the semantic map data before the update. The semantic map data before the update records object information for each second object among items 101 to 106, and the object information for each second object includes the second object features and second point cloud data of that second object. The semantic map data update device 200 updates the semantic map data by updating the data of the second object. Figure 3 The processing of the first image sequence of the image shown in (a) and the depth image corresponding to each first image in the first image sequence can update the semantic map data.

[0143] Based on the first image sequence and the depth image sequence, the first object determined by the semantic map data update device 200 can be respectively... Figure 3 Items 311 to 316 are recorded in the image shown in (a).

[0144] The update module 240 can determine, through steps S711 to S713, that the candidate second objects include items 101 to 106, and determine that item 104 (cabinet) among the candidate second objects is a disappearing object, while the other candidate second objects are not disappearing objects. The update module 240 can delete the second point cloud data of item 104 from the semantic map data.

[0145] The update module 240 can also determine, through steps S721 to S724, that items 101 to 103, 105 and 106 in the semantic map data are target second objects that match the object features of items 311 to 313, 315 and 316 in the first object.

[0146] If the semantic map data does not include a second object that matches the object features of item 314, then for a first object whose object features do not match any of the second object's features in the semantic map data (i.e., a first object not recorded in the semantic map data), the update module 240 can add the object information of that first object. The update module 240 can add the object information of item 314 to the semantic map data. Item 314 can be understood as a new second object in the updated semantic map data. The object information of item 314 can include its point cloud data and object features, as well as its semantic label. The update module 240 can also assign an identifier to the newly added object and use this identifier as part of the object information added to the semantic map data.

[0147] If the semantic map data includes a second object that matches the object features of item 314, and the semantic map data does not include the second point cloud data of the second object, then the first point cloud data of item 314 can be used as the second point cloud data of the second object.

[0148] For a first object that matches the object features of a second object in the semantic map data (i.e., a first object already recorded in the semantic map data), the update module 240 can calculate the overlap degree of the point cloud data. The update module 240 can determine that the overlap degree between the first point cloud data and the second point cloud data of items 101 (door), 102 (wall), 103 (bed), and 105 (ground) is greater than or equal to the overlap degree threshold. Therefore, the second point cloud data of items 101, 102, 103, and 105 in the semantic map data does not require further processing. The update module 240 can determine that the overlap degree between the second point cloud data of item 106 (chair) in the semantic map data and the first point cloud data of item 316 (i.e., item 106) determined based on the first image sequence is less than the overlap degree threshold. Therefore, the second point cloud data of item 106 in the semantic map data is replaced with the first point cloud data of item 316 determined based on the first image sequence.

[0149] After steps S711 to S713 and steps S721 to S724 are performed in update module 240, the updated semantic map data can be obtained. The semantic map represented by the updated semantic map data is as follows: Figure 8 As shown.

[0150] When electronic devices run repeatedly in the same environment, the semantic map data update device 200 provided in this application embodiment can continuously maintain and update the item information in the semantic map data, avoiding the need to construct semantic map data multiple times for the same environment. The semantic map data can accurately reflect changes in objects in the environment, enabling real-time environmental perception. The semantic map data update device 200 updates the semantic map data at the object level. Therefore, the semantic map data maintained by the semantic map data update device 200, in addition to enabling flexible path planning, can also support object-level navigation and grasping tasks due to its more accurate environmental perception. This allows for complex human-computer interaction tasks targeting specific objects, making it suitable for more complex tasks and application scenarios, and providing convenience for users.

[0151] In the semantic map data update device 200, the object recognition model 210, the segmentation and tracking module 220, and the object feature extraction model 250 may include neural network models obtained through supervised training.

[0152] A supervised training model can be trained on training data, which includes training samples and sample labels. During supervised training, the initial model is processed based on the training samples to obtain training information. The parameters of the initial model can be adjusted according to the difference between the training information and the sample labels. This difference can be understood as the loss value. The adjusted initial model is the trained neural network model.

[0153] For object recognition models, training samples can be training images, and sample labels can include the category of the object in the training image and the location information of the recognition box it belongs to.

[0154] For the segmentation tracking model 220, the image object segmentation model, the training samples may include a first training image and training cue information. The training cue information may be the location information of the bounding boxes of at least one training object recorded in the first training image. The sample labels may be the contours of each training object among the at least one training object recorded in the first training image.

[0155] In some embodiments, the training of an image segmentation model can be divided into three stages. In the first stage, the training data used in the training process includes training prompts and sample labels representing the primary (or prominent) objects in the training segmentation image, and the sample labels can be manually annotated. In the second stage, the training data used in the training process includes training prompts representing secondary (or non-prominent) objects in the training segmentation image. The contours of the training objects in the sample labels can be manually modified based on the contours obtained by processing the training samples from the second stage using the model adjusted in the first stage. The second stage can be understood as a semi-automatic stage. In the third stage, the training samples are processed multiple times using the model adjusted in the second stage. From the contours obtained each time the model adjusted in the second stage processes the samples, stable contours are selected as the sample labels for the third stage. The third stage can be understood as a fully automatic stage. Traditional neural network model training requires a large amount of labeled data. Training the image segmentation model through three stages reduces the amount of labeled data required and lowers manual labor costs.

[0156] The image object segmentation model can be a SAM model. Multiple training datasets can be used during the training of the SAM model. In each training dataset, training prompts can include one or more of the following: training description text, training description regions, and training description points. The training description text describes the training object. The training description regions can include the bounding box position and / or initial contour of the training object in the second training image. The accuracy of the initial contour of the training object in the second training image is lower than the accuracy of the contour of the training object in the sample labels. The training prompts in these multiple training datasets include training description text, training description regions, and training description points, and the training prompts in these multiple training datasets include both training description regions representing the bounding box position of the training object in the second training image and training description regions representing the initial contour of the training object in the second training image. Training description points can include one or more points located on the training object in the second training image.

[0157] In segmentation tracking model 220, the video object segmentation model used to segment objects in the video can also be a neural network model. For the video object segmentation model, training samples may include a first training image sequence and training cue information. The first training image sequence may include multiple third training images. The training cue information may include the position information of the bounding box of the training object in each third training image that records the training object. For the video object segmentation model, the sample label may be the contour of the training object in each third training image that records the training object in the first training image sequence.

[0158] In some embodiments, for the object feature extraction model 250, training samples may include a second training image sequence. The second training image sequence includes multiple fourth training images. Each fourth training image records a training object. The sample label is a feature of the training object. During the training of the object feature extraction model 250, the amount of training data used may be one or more. For different training objects, the difference between the training features in the sample labels is greater than or equal to a difference threshold.

[0159] The training samples may also include region information of the area where the training object is located in the fourth training image. The region information of the area where the training object is located can represent the outline of the training object and / or the position information of the bounding box where the training object is located in the fourth training image.

[0160] In other embodiments, for the object feature extraction model 250, training samples may include a fifth training image. The sample label is the training object recorded in the fifth training image. Training samples may also include region information of the area where the training object is located in the fifth training image. The region information of the area where the training object is located may represent the outline of the training object in the fifth training image and / or the position information of the bounding box where the training object is located.

[0161] Figure 2 The semantic map data update device 200 shown can be located in an electronic device. The electronic device can be a robot. The electronic device can perform... Figure 9 The method shown.

[0162] Figure 9 This is a schematic flowchart illustrating a system operation method provided in an embodiment of this application. Method 800 can be understood as a method for performing simultaneous localization and mapping (SLAM). Through the SLAM algorithm, the position and orientation of an electronic device in the environment can be determined based on sensor data, and a semantic map of the environment can be constructed. Method 800 includes steps S801 to S806.

[0163] Step S801: Start running.

[0164] For example, an electronic device can start operating when it detects a power-on signal input by the user via the power button.

[0165] Step S802: Determine whether semantic map data is recorded.

[0166] If it is determined in step S802 that semantic map data is recorded, proceed to step S803.

[0167] Step S803: Determine the coordinates of the electronic device in the map represented by the semantic map data.

[0168] Electronic devices can acquire images and determine their coordinates within a map represented by semantic map data based on the acquired images. For example, the location of the electronic device on the map can be determined by identifying the semantic features of objects in the acquired image and matching them with the features of a second object in the semantic map data. Alternatively, features can be extracted from the acquired image to obtain multiple feature points in the image to be processed, and these feature points can be matched with feature points in the semantic map data to determine the location of the electronic device on the map based on the matching results.

[0169] Besides visual positioning, other positioning technologies, such as the Global Navigation Satellite System (GNSS), can be used to determine the coordinates of electronic devices within a map represented by semantic map data. Semantic map data can include the location information of multiple points within the map.

[0170] If it is determined in step S802 that no semantic map data has been recorded, proceed to step S804.

[0171] Step S804: Establish initial semantic map data.

[0172] Establishing initial semantic map data can be understood as creating a coordinate system with the location of the electronic device as the origin. The initial semantic map data does not include object features or point cloud data.

[0173] After step S803 or step S804, steps S805 and S806 can be performed.

[0174] Step S805: Acquire image sequences and depth image sequences, and record the camera parameters of the electronic device when acquiring each frame of the image sequence.

[0175] Step S806: Update the semantic map data based on the image sequence, depth image sequence, and camera parameters.

[0176] The semantic map data can be updated via a semantic map data updating device 200 in an electronic device. The electronic device can operate in indoor or outdoor environments.

[0177] In outdoor environments, electronic devices have a wider range of movement, reducing the likelihood of them reverting to the same location. Semantic map data requires significant storage space. To reduce storage space consumption, electronic devices can delete previously stored semantic map data upon reaching a new location. Even when operating at the same location, vehicles, people, and objects in the vicinity of an outdoor electronic device may change frequently. Compared to outdoor environments, indoor environments have lower frequency of object location changes, resulting in a higher demand for updating semantic map data.

[0178] When depth information is determined through optical principles, if Time-of-Flight (ToF) technology is used to detect depth—that is, calculating distance by measuring the round-trip time of a light pulse—the light signal attenuates significantly at long distances, leading to increased time measurement errors. Simultaneously, increased ambient light interference affects signal quality. Conversely, if stereo vision technology relying on the parallax principle of binocular or multi-view cameras is used to detect depth, the parallax decreases as the object is farther away, resulting in lower depth resolution and increased depth estimation errors. In indoor scenes, due to the limited space, the distance represented by depth information acquired by electronic devices is generally smaller. Therefore, the accuracy of depth information is higher in indoor environments, leading to higher accuracy in the updated semantic map data.

[0179] An electronic device including the semantic map data update device 200 can perform [operations] via the semantic map data update device 200. Figure 10 The method shown is used to perform step S806.

[0180] Figure 10 This is a schematic flowchart of a semantic map data update method provided in an embodiment of this application. Figure 10 The method shown includes steps S910 to S940. The execution subject of the method provided in this application embodiment can be an electronic device, or a software / hardware module in the electronic device capable of image acquisition. For ease of explanation, the following embodiments use an electronic device as an example.

[0181] Step S910: Obtain the first image sequence and the depth image corresponding to each frame of the first image in the first image sequence.

[0182] In other words, in step S910, a first image sequence and a depth image sequence can be obtained. The first image in the first image sequence corresponds one-to-one with the depth image in the depth image sequence. Any first image in the first image sequence and its corresponding depth image are obtained by capturing the same scene.

[0183] The first image sequence and the depth image sequence can be obtained by receiving data sent from other devices. Alternatively, the first image sequence and the depth image sequence can be obtained by electronic devices through data acquisition and processing.

[0184] Images are captured by a camera in an electronic device. Based on the camera's intrinsic parameters, distortion correction is performed on the captured images to obtain a first image sequence. Camera intrinsic parameters, also known as intrinsic parameters of the camera, include distortion coefficients, such as radial distortion coefficients and tangential distortion coefficients. A distortion correction algorithm adjusts the pixel positions according to the distortion coefficients, correcting distortion in each frame to obtain the first image sequence. The images captured by the camera can be either color or grayscale. That is, the first image in the first image sequence can be either a color image or a grayscale image.

[0185] Depth image sequences can be determined using infrared structured light, time-of-flight methods, or binocular stereo vision. In other words, depth image sequences can be determined by distance sensors or based on images captured by multiple cameras.

[0186] Step S920: Determine the first object features of the first object recorded in the first image sequence based on the first image sequence.

[0187] The first object can be understood as the object recorded in the first image sequence. In the first image sequence, there can be one or more first objects. Based on the first image sequence, the first object characteristics of each first object can be determined.

[0188] Dissimilar objects can be understood as objects that differ in appearance. That is, objects that differ in shape, color, size, texture, etc., are considered dissimilar objects. Objects belonging to the same category can be identical or different. For example, two water glasses belong to the same category—both are water glasses—but their colors and shapes may differ, making them dissimilar objects.

[0189] For different objects, their object characteristics can be different. Therefore, based on the similarity between two object characteristics, it can be determined whether the objects corresponding to those two object characteristics are the same object.

[0190] Step S930: Determine the first point cloud data of the first object based on the first image sequence and the depth image corresponding to each frame of the first image.

[0191] The number of first object images for each first object can be one or more. Based on the first image sequence, the object contour of the first object in the first object images recording the first object can be determined. Based on the object contour of the first object in each first object image and the corresponding depth image for each first object image, the first point cloud data of the first object can be determined.

[0192] The object outline in an image is the boundary line between the object and its background, typically composed of a series of consecutive pixels. These pixels are usually located at the boundary between the object and the background and have high gradient values ​​(i.e., significant changes in grayscale or color). By detecting abrupt changes in grayscale or color in the image, the boundary of the object can be determined, that is, the object outline.

[0193] Alternatively, an object segmentation model can be used to process the first image sequence to obtain the object contour of each first object in each image containing that first object. The object segmentation model is used to segment objects in the image sequence. The object segmentation model can be a neural network model. The determination of the first object's point cloud data by the object segmentation model can be based on the object contour of the first object in the first object image.

[0194] In some embodiments, in step S920, after determining the object outline of the first object in each first object image, the first object feature of the first object can be determined based on the first object image and the object outline of the first object in the first object image.

[0195] For example, based on the object outline of the first object in the first object image, the image of the first object located within the object outline can be determined. Based on the image of the first object located within the object outline of the first object, the first object features of the first object can be determined. Determining the first object features of the first object based on the image located within the object outline of the first object in the first object image avoids the influence of other objects outside the first object on the first object features, thus improving the accuracy of the first object features.

[0196] For a given first object, if there are multiple images of the first object, feature extraction can be performed on the object outline of the first object in any one of the first object images to obtain the first object feature. In other words, the first object feature can be obtained based on the features of the first object in any one of the first object images.

[0197] When there are multiple images of the first object, features of the first object located within the object contour in each of the first object images can be extracted separately to obtain the features of the first object in each first object image. The features of the first object in each first object image can then be fused to obtain the first object feature of the first object.

[0198] When there are multiple first object images, feature extraction can be performed on a second image sequence that includes images of the object contour of the first object in each of the first object images to obtain the first object features of the first object. That is, each image in the second image sequence can be an image located within the object contour of the first object in a first object image.

[0199] In other embodiments, during the processing of the first image sequence to determine the object contour of the first object in each first object image, first object features of the first object can be obtained. Based on the first object features, the object contour of the first object in the first object image can be determined.

[0200] In the process of determining the first point cloud data, the object contour of the first object in the first object image is used. The first object feature of the first object can be obtained during the process of determining the object contour of the first object in the first object image. Therefore, when performing object segmentation on the first image sequence, the first object feature of the first object is determined simultaneously with determining the object contour of the first object in the first object image of the first image sequence. Determining the first object feature of the first object requires no additional operations, improving the processing efficiency of the semantic map data update method.

[0201] Before performing object segmentation processing on the first image sequence using an object segmentation model to determine the contour of the first object in the first object image, object recognition can be performed on each frame of the first image sequence. By recognizing the object in the first image, it can be determined whether the first image contains the first object, i.e., whether the first image for which object recognition is performed is a first object image containing the first object, thereby identifying the first object image in the first image sequence. Object recognition in the first image can also determine the positional information of the first object within the first object image.

[0202] The positional information of the first object in the first object image can be represented as the positional information of the region where the first object is located in the first object image. Alternatively, the positional information of the first object in the first object image can be represented as the positional information of at least one point, which can be a point on the first object in the first object image. Object recognition in the first object image can, for example, obtain the positional information of the recognition box containing the first object. The shape of the recognition box can be preset, such as a rectangle. The positional information of the recognition box can be represented by the coordinates of multiple vertices of the recognition box, or by the coordinates of the center point of the recognition box, as well as the length and width of the recognition box.

[0203] Based on the position information of the first object in the first object image, the object outline of the first object included in the first object image in the first image sequence can be determined by performing object segmentation processing on the first image sequence.

[0204] By performing object segmentation processing on the first image sequence based on the position information of the recognition box where the first object is located in the first object image, the object segmentation result of the first image sequence, i.e. the object outline of the first object, can be more accurate.

[0205] In some other embodiments, a first object feature of the first object can be determined based on the first object image and the position information of the first object within the first object image. For example, the first object feature of the first object can be determined based on an image located within the recognition box of the first object in the first object image.

[0206] When there are multiple first object images, the first object feature can be obtained by extracting features from the image within the recognition box of a single first object image, or by extracting features from the images within the recognition boxes of multiple first object images and then fusing the extracted features. Alternatively, the first object feature can also be obtained by extracting features from a third image sequence that includes images within the recognition boxes of each first object image. That is, each image in the third image sequence can be an image located within the recognition box of a first object image.

[0207] It should be understood that the first object features of the first object can also be determined based on the first object image, the position information of the recognition box in the first object image, and the outline of the first object in the first object image.

[0208] After determining the outline of the first object image, the first point cloud data of the first object can be determined based on the depth image corresponding to the first object image and the outline of the first object in the first object image.

[0209] For example, in step S910, pose information corresponding to each first image in the first image sequence can also be obtained. The pose information corresponding to each first image can be understood as the pose information of the electronic device acquiring the first image when acquiring the first image. The pose information of the electronic device can represent the position and orientation of the electronic device. Based on the depth image corresponding to the first object image, the pose information corresponding to the first object image, and the contour of the first object in the first object image, a single frame point cloud data of the first object can be determined.

[0210] If the first image and the corresponding depth image are obtained from different sensors, in step S910, the relative positional relationship between the camera used to acquire the first image and the sensor used to acquire the depth image in the electronic device can also be obtained. Based on the depth image corresponding to the first object image, the pose information corresponding to the first object image, the contour of the first object in the first object image, and the relative positional relationship between the camera used to acquire the first object image and the sensor used to acquire the depth image, a single frame point cloud data of the first object can be determined.

[0211] In the first image sequence, if there is only one image of a first object, the single-frame point cloud data of the first object can be used as the first point cloud data of the first object. If there are multiple images of a first object in the first image sequence, the multiple single-frame point cloud data of the first object can be fused to obtain the first point cloud data of the first object. The number of multiple single-frame point cloud data of the first object can be equal to the number of images of the first object.

[0212] Step S940: Based on the second object features of the second object in the semantic map data, the similarity between the features of the first object, and the first point cloud data, update the semantic map data. The similarity between the object features of different objects is less than the similarity threshold.

[0213] Semantic map data includes second object features for each of at least one second object. During the updating of semantic map data, it can be determined whether the semantic map data records a target second object, that is, whether the at least one second object in the semantic map data includes a target second object. A target second object is a second object in the semantic map data whose second object features have a similarity greater than a similarity threshold with the first object features.

[0214] If the semantic map data does not include the target second object, the features of the first object and the first point cloud data can be added to the semantic map data. In other words, if the similarity between the features of the second object and the features of the first object in the semantic map data is less than or equal to a similarity threshold, it can be determined that the first object is not recorded in the semantic map data. Therefore, the first object can be treated as a new second object, its features can be used as the second object features of the new second object, and the first point cloud data of the first object can be used as the second point cloud data of the new second object, and these can be added to the semantic map data.

[0215] In indoor environments, especially in home settings, changes in the location of an object not recorded in the semantic map data—such as newly placed items by the user—can lead to improvements. Adding the object's first object features and initial point cloud data to the semantic map data makes the updated semantic map more accurate.

[0216] When the semantic map data includes a second target object, it can be determined that the second target object in the semantic map data is the same object as the first object, thereby determining the second point cloud data of the second target object in the semantic map data.

[0217] In indoor environments, especially in home settings, a change in the location of an object recorded in semantic map data usually indicates that the user has moved the object. Conversely, if the object's location remains unchanged, it usually means the user has not moved it. When the semantic map data includes a target second object (i.e., the first object is already recorded in the semantic map data), the second point cloud data of the target second object is updated based on the first point cloud data of the first object, making the updated semantic map data more accurate.

[0218] The semantic map data may include second point cloud data for each second object. Alternatively, the semantic map data may include second point cloud data for a subset of the second objects.

[0219] If the semantic map data does not include the second point cloud data of the target second object, the first point cloud data based on the first object can be added to the semantic map data as the second point cloud data of the target second object. If the semantic map data includes the second point cloud data of the target second object, the second point cloud data of the target second object in the semantic map data can be updated based on the first point cloud data of the first object.

[0220] Updating the second point cloud data of a target second object in the semantic map data can be achieved by replacing the second point cloud data of the target second object with the first point cloud data. In other words, the updated second point cloud data of the target second object is the first point cloud data of the first object.

[0221] Alternatively, during the update of the second point cloud data of the target second object in the semantic map data, it can be determined whether the overlap between the second point cloud data of the target second object and the first point cloud data of the first object is less than an overlap threshold. If the overlap between the second point cloud data of the target second object and the first point cloud data of the first object is greater than or equal to the overlap threshold, the second point cloud data of the target second object can be replaced with the first point cloud data. If the overlap between the second point cloud data of the target second object and the first point cloud data of the first object is less than the overlap threshold, the first point cloud data of the first object and the second point cloud data of the target second object can be fused to obtain the updated second point cloud data of the target second object.

[0222] When the overlap between the second point cloud data of the second target object and the first point cloud data of the first object is high, greater than or equal to the overlap threshold, the updated second point cloud data of the second target object is the fusion result of the first point cloud data of the first object and the second point cloud data of the second target object. When there is a large overlap between the second and first point cloud data, the information from the two point cloud data is merged to compensate for the deficiencies of a single data set. Generally, the first and second point cloud data are obtained from video sequences from different perspectives. Multi-view point cloud data fusion can fill the gaps in the incompleteness of point cloud data caused by perspective limitations, making the obtained point cloud data more accurate and complete in reflecting the shape of the object.

[0223] If the second target object is the same as the first object, but the overlap is low (below the overlap threshold), it can be determined that the pose of the second target object has changed. That is, the pose of the second target object is different from the pose when the point cloud data and feature information of the second target object were added to the semantic map data; it is not the original pose. If we want to fuse the point cloud data of the object before and after the pose change (i.e., fuse the first point cloud data and the second point cloud data), point cloud registration is required first. Point cloud registration refers to the process of aligning 3D point cloud data from different coordinate systems to the same coordinate system through geometric transformations (rotation, translation), aiming to achieve the fusion of multi-view point clouds and the complete reconstruction of the 3D scene.

[0224] In semantic map data, the second point cloud data of a second object may not reflect the object's complete shape, and the first point cloud data of the first object obtained from the first image sequence and depth image sequence may also not fully reflect the object's shape. Furthermore, the image sequence used to determine the second point cloud data of the second object in the semantic map data may have a different orientation than the object in the first image sequence. In other words, the second point cloud data of the second object in the semantic map data and the first point cloud data of the first object may record the same object with different orientations. The incompleteness of the first and second point cloud data, as well as the difference in orientation, both make point cloud registration more difficult and result in lower registration accuracy.

[0225] Therefore, if the target second object and the first object are the same object, but with low overlap, the second point cloud data of the target second object is registered with the first point cloud data to obtain new point cloud data, which is then added to the semantic map data. The error generated in each registration process accumulates in the point cloud data of that object in the semantic map data, resulting in low accuracy of the point cloud data of that object.

[0226] Therefore, when the target second object and the first object are the same object but have a low degree of overlap, replacing the target second object's second point cloud data with the first point cloud data of the first object can avoid the error caused by the registration of the first point cloud data and the second point cloud data being recorded in the point cloud data, thus improving the accuracy of the point cloud data.

[0227] In other words, the determination of the second point cloud data of the target second object can be based on the overlap condition. The overlap condition is: the semantic map data includes the second point cloud data of the target second object, and the overlap degree between the second point cloud data of the target second object and the first point cloud data is greater than or equal to the overlap degree threshold.

[0228] If the semantic map data includes a second target object, and the second target object meets the overlap condition, the second point cloud data of the second target object can be fused with the first point cloud data to obtain new second point cloud data for the second target object. If the semantic map data includes the second target object, but the second target object does not meet the overlap condition, the first point cloud data is used as the second point cloud data for the second target object. Determining the second point cloud data of the second target object based on the overlap condition can improve the accuracy of the second point cloud data in the semantic map data.

[0229] Based on the objects recorded in the first image sequence, updating the semantic map data includes not only adding new items and updating the point cloud data of recorded objects based on pose changes, but also deleting the point cloud data of missing objects. A missing object can be understood as a second object recorded in the semantic map data that, according to the semantic map data, should have appeared in the first image sequence but did not actually appear.

[0230] At least one candidate second object can be identified from at least one second object in the semantic map data. The second point cloud data of each candidate second object is located within the view frustum corresponding to at least one frame of the first image in the first image sequence and is not occluded. A candidate second object can be understood as a second object that should appear in the first image sequence according to the semantic map data.

[0231] The similarity between the second object features and the first object features of the candidate second object can be calculated to determine whether the candidate second object is a missing object. A missing object can be a second object whose similarity between the second object features and the first object features is less than or equal to the similarity threshold. If the candidate second object includes a missing object, the second point cloud data of the missing object can be deleted from the semantic map data.

[0232] By identifying unoccluded candidate second objects within the view frustum corresponding to the first image from the second objects in the second point cloud data, and deleting the second point cloud data of the candidate second objects from the semantic map data when the similarity between the second object features and the first object features of the candidate second objects is less than a similarity threshold, the updating of the semantic map data can be made more accurate.

[0233] When the second point cloud data of a disappearing object is deleted, object information such as the second object's features can also be deleted. Alternatively, when the second point cloud data of a disappearing object is deleted, object information such as the second object's features can still be retained in the semantic map data.

[0234] Without deleting object information such as the second object features of the disappeared object, when the semantic map data is updated again based on the new first image sequence, the semantic map data may not include the object features of some second objects, but it may not include the second point cloud data of the second object.

[0235] Before calculating the comparison between the first object features of the first object and the second object features of the second object, the category to which each first object belongs can be determined. The comparison between the first object features of the first object and the second object features of the second object can be performed by calculating the similarity of object features between first objects and second objects of the same category.

[0236] In other words, the semantic map data can record the category of each second object. Based on the category of each second object in the semantic map data, at least one second object in the semantic map data is identified that shares the same category as the first object. The similarity between the features of the second object and the features of the first object is calculated, and the semantic map data can be updated based on the similarity between the features of the second object and the features of the first object, as well as the first point cloud data of the first object.

[0237] It should be understood that when there are multiple first objects, for each first object, a second object of the same category in the semantic map data can be identified as belonging to the same category as that first object.

[0238] A second object that is identical to the first object belongs to the same category as the first object. By calculating the similarity of object features between the first and second objects of the same category, it can be determined whether a second object identical to the first object is recorded in the semantic map data. This eliminates the need to perform similarity calculations on object features for first and second objects of different categories, reducing computational load and improving processing efficiency.

[0239] The category of the first object can be determined by performing object recognition on each frame of the first image. That is, performing object recognition on each frame of the first image not only determines the location information of the recognition box containing the first object in the first object image, but also determines the category of the first object within each recognition box in the first object image. The category of the first object within each recognition box can also be understood as the category corresponding to that recognition box.

[0240] If the semantic map data includes the category of a second object, and it is determined that the semantic map does not include the target second object (i.e., it does not include objects identical to the first object), then in addition to adding the first point cloud data of the first object to the semantic map data, the category of the first object can also be added to the semantic map data. Conversely, if it is determined that the semantic map data includes a disappearing object, in addition to deleting the second object features and second point cloud data of the disappearing object from the semantic map data, the category of the disappearing object can also be deleted from the semantic map data.

[0241] The semantic map data update method provided in this application updates the semantic map data based on the features of a first object recorded in a first image sequence. The similarity between the features of different first objects is less than a similarity threshold; that is, based on the features of the first object, it can be determined whether the first object is the same as the object recorded in the semantic map data. Therefore, the update of the semantic map data is an object-level update, rather than a category-level update, resulting in more accurate semantic map updates.

[0242] When the similarity between the object features of different objects is less than a similarity threshold, the semantic map data includes the object features of the second object, thus achieving an accurate representation of objects in the environment. The semantic map data is updated based on whether the same object is recorded in it. Therefore, over time, the existing semantic map data is updated based on newly acquired image sequences, eliminating the need to repeatedly create semantic map data for the same location and reducing the consumption of processing resources.

[0243] Electronic devices can be used to perform Figure 9 The system operation method shown or Figure 10 The semantic map data update method is shown below. The following section combines... Figure 11 The structure of the electronic device is explained.

[0244] Figure 11 A hardware system for an electronic device applicable to this application is shown.

[0245] The method provided in this application can be applied to various electronic devices capable of image acquisition, such as mobile phones, tablets, wearable devices, laptops, netbooks, personal digital assistants (PDAs), and in-vehicle devices. This application does not impose any restrictions on the specific type of electronic device.

[0246] Figure 11A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 151, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0247] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0248] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0249] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0250] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0251] The device 100 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 194, and application processor.

[0252] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0253] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard red-green-blue (RGB), YUV, or other image signal format. In some embodiments, device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0254] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when device 100 selects a frequency point, the DSP can perform Fourier transforms on the frequency energy.

[0255] Video codecs are used to compress or decompress digital video. Device 100 may support one or more video codecs. Thus, device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0256] An NPU (Neural Processing Unit) is a processor that borrows from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, to quickly process input information and continuously learn. Through an NPU, device 100 can achieve intelligent cognitive functions, such as object recognition, object segmentation, facial recognition, speech recognition, and text understanding.

[0257] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0258] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0259] The accelerometer 180E can detect the magnitude of acceleration of device 100 in various directions (generally the x-axis, y-axis, and z-axis directions that are perpendicular in a three-dimensional Cartesian coordinate system). When device 100 is stationary, it can detect the magnitude and direction of gravity. The accelerometer 180E can also be used to identify the attitude of device 100, serving as input parameters for applications such as screen orientation switching and pedometers.

[0260] Distance sensor 180F is used to measure distance. Device 100 can measure distance via infrared or laser. In some embodiments, such as in a shooting scenario, device 100 can utilize distance sensor 180F for distance measurement to achieve fast focusing.

[0261] The button 190 can be a mechanical button or a touch button. The device 100 can receive button input signals and implement functions related to the button input signals. The button 190 includes a power button and volume buttons, etc.

[0262] This application provides a thread scheduling apparatus, including various functional units for executing the thread scheduling method provided in this application. The term "unit" here can be implemented in software and / or hardware, and is not specifically limited thereto.

[0263] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0264] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0265] This application also provides a chip, which includes a data interface and one or more processors. When the one or more processors execute instructions, they read instructions stored in memory through the data interface to implement the thread scheduling method described in the above method embodiments.

[0266] The one or more processors can be general-purpose processors or special-purpose processors. For example, the one or more processors can be central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0267] The chip can be used as a component of a terminal device or other electronic device. For example, the chip can be located in electronic device 100.

[0268] Processors and memory can be configured separately or integrated together. For example, processors and memory can be integrated onto a system-on-a-chip (SoC) in a terminal device. That is, the chip can also include memory.

[0269] The memory can store programs, which can be run by the processor to generate instructions, causing the processor to execute the thread scheduling method described in the above method embodiments according to the instructions.

[0270] Optionally, the memory may also store data. Optionally, the processor may also read data stored in the memory, which may be stored at the same memory address as the program, or the data may be stored at a different memory address than the program.

[0271] This chip can be installed in electronic devices.

[0272] For example, the memory can be used to store the relevant program of the thread scheduling method provided in the embodiments of this application, and the processor can be used to call the relevant program of the thread scheduling method stored in the memory to implement the thread scheduling method of the embodiments of this application.

[0273] For example, a first image sequence and a depth image corresponding to each frame of the first image sequence are obtained, wherein any frame of the first image sequence and the depth image corresponding to the first frame of the first image sequence are obtained by collecting data from the same scene; based on the first image sequence, the first object feature of the first object recorded in the first image sequence is determined; based on the first image sequence and the depth image corresponding to each frame of the first image, the first point cloud data of the first object is determined; based on the similarity between the second object feature of the second object in the semantic map data and the first object feature, and the first point cloud data, the semantic map data is updated, wherein the similarity between the object features of different objects is less than a similarity threshold.

[0274] This application also provides a computer program product that, when executed by a processor, implements the thread scheduling method described in any of the method embodiments of this application.

[0275] The computer program product can be stored in memory, for example, it is a program. The program is eventually converted into an executable object file that can be executed by the processor after processes such as preprocessing, compilation, assembly and linking.

[0276] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the thread scheduling method described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0277] The computer-readable storage medium is, for example, memory. Memory can be volatile or non-volatile, or it can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0278] In the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance, or a specific order or sequence. Those skilled in the art will understand the specific meaning of these terms in this application based on the specific circumstances.

[0279] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0280] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0281] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0282] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0283] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0284] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0285] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A semantic map data update method, characterized in that, The method includes: A first image sequence is obtained, and a depth image corresponding to each frame of the first image in the first image sequence is obtained by collecting the same scene. Based on the first image sequence, determine the first object feature of the first object recorded in the first image sequence; The first point cloud data of the first object is determined based on the first image sequence and the depth image corresponding to each frame of the first image; The semantic map data is updated based on the similarity between the second object features of the second object and the first object features in the semantic map data, as well as the first point cloud data. The similarity between the object features of different objects is less than the similarity threshold.

2. The method according to claim 1, characterized in that, The step of determining the first object feature of the first object recorded in the first image sequence based on the first image sequence includes: The object outline of the first object included in the first object image in the first image sequence is determined, and the first object image is a first image that records the first object; The features of the first object are determined based on the object outline of the first object and the first object image in which the first object is located.

3. The method according to claim 2, characterized in that, Determining the object outline of the first object included in the first object image in the first image sequence includes: Perform object recognition on each frame of the first image in the first image sequence to determine the position information of the first object in the first object image; Based on the position information of the first object in the first object image, the object outline of the first object is determined by performing object segmentation processing on the first image sequence.

4. The method according to any one of claims 1-3, characterized in that, The step of updating the semantic map data based on the similarity between the second object features of the second object and the first object features in the semantic map data, and the first point cloud data, includes: If the target second object is not included in the second object included in the semantic map data, the first object feature and the first point cloud data are added to the semantic map data, and the similarity between the second object feature of the target second object and the first object feature is greater than or equal to the similarity threshold. If the target second object is included in the second object included in the semantic map data, the second point cloud data of the target second object in the semantic map data is determined based on the first point cloud data.

5. The method according to claim 4, characterized in that, When the target second object is included in the second object included in the semantic map data, determining the second point cloud data of the target second object in the semantic map data based on the first point cloud data includes: If the semantic map data includes the target second object and the target second object meets the overlap condition, the second point cloud data of the target second object is fused with the first point cloud data to obtain new second point cloud data of the target second object. The overlap condition is: the semantic map data includes the second point cloud data of the target second object, and the overlap degree between the second point cloud data of the target second object and the first point cloud data is greater than or equal to the overlap degree threshold. If the semantic map data includes the target second object, and the target second object does not meet the overlap condition, the first point cloud data is used as the second point cloud data of the target second object.

6. The method according to claim 5, characterized in that, The method further includes: Among the second objects included in the semantic map data, candidate second objects are determined. The candidate second objects are located in the frustum range corresponding to at least one frame of the first image in the first image sequence and are not occluded. If the candidate second object includes a missing object, the second point cloud data of the missing object is deleted from the semantic map data, and the similarity between the second object feature of the missing object and the first object feature is less than or equal to the similarity threshold.

7. The method according to any one of claims 1-6, characterized in that, The semantic map data also includes the category of each second object. Updating the semantic map data based on the similarity between the second object features and the first object features in the semantic map data, and the first point cloud data, includes: Based on the category of the second object in the semantic map data, determine a second object of the same category as the first object; Calculate the similarity between the features of the second object and the features of the first object in the same category; The semantic map data is updated based on the similarity between the features of the second object and the features of the first object in the same category, as well as the first point cloud data.

8. An electronic device, characterized in that, The device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to retrieve and run the computer program from the memory, causing the electronic device to perform the method of any one of claims 1 to 7.

9. A chip, characterized in that, It includes a processor and a data interface, wherein the processor reads instructions stored in memory through the data interface to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for implementing the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, When the computer program product is run on an electronic device, the electronic device is used to perform the method of any one of claims 1-7.