Image processing method, device and storage medium based on open vocabulary

Abstract

The present application discloses an image processing method, device and storage medium based on open vocabulary, which relates to the technical field of three-dimensional mapping. The method includes: based on the current frame collected by the robot, obtaining the instance area and instance name corresponding to each first instance object, and inputting them into the large language model to obtain the position relationship description text and object attribute description text of the first instance object; obtaining the local point cloud map and the global point cloud map of the current frame; searching for the first target instance object matching any first instance object in the global point cloud map to perform point cloud merging operation and text merging operation, and updating the global point cloud map; the point cloud merging operation is used to merge the point clouds of the instance objects; the text merging operation is used to merge the position relationship description text, object attribute description text and instance name of the instance objects respectively. In this way, the mapping accuracy can be improved, and the environment can be parsed into a semantically rich and structured three-dimensional scene map.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional mapping technology, and in particular to an open vocabulary-based image processing method, device and storage medium. Background Art

[0002] 3D mapping is a process of creating a model with 3D data by using 3D production software. This process can make a 2D plane three-dimensional, making it more intuitive and three-dimensional. During the 3D mapping process, the robot continuously collects multiple frames of RGB-D images (including photos of depth maps), each frame of which can be converted into 3D point cloud data within the image area, and the global 3D point cloud map is constructed and updated through continuous multi-frame point clouds. However, in the 3D mapping process, the existing technology is prone to ghosting problems, resulting in low mapping accuracy and effect. Moreover, usually without limiting the category of instance objects, it is impossible to understand the objects in a complex environment and the relationship between objects, and lacks a structural understanding of the scene. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an open vocabulary-based image processing method, device and storage medium to solve the technical problems of three-dimensional mapping ghosting, low precision and lack of scene structured understanding in the prior art.

[0004] In order to achieve the above-mentioned object, the first aspect of the present application provides an image processing method based on an open vocabulary, comprising:

[0005] Obtaining a current frame collected by the robot, and inputting the current frame into an instance segmentation model to obtain an instance region and an instance name corresponding to each first instance object;

[0006] Acquire a local point cloud map corresponding to the target site in the current frame based on a preset algorithm, where the local point cloud map includes first point clouds corresponding to a plurality of first instance objects;

[0007] Inputting all instance regions and all instance names into the first language model and the second language model respectively, so as to obtain a position relationship description text and an object attribute description text for determining each first instance object respectively;

[0008] Acquire a global point cloud map for the target site, the global point cloud map including second point clouds corresponding to multiple second instance objects, historical position relationship description text, historical object attribute description text, and historical instance names;

[0009] For any first instance object, searching for a first target instance object matching the first instance object among a plurality of second instance objects included in the global point cloud image;

[0010] For any first instance object, a point cloud merging operation and a text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map; wherein the point cloud merging operation is used to merge the first point cloud corresponding to the any first instance object into the second point cloud corresponding to the first target instance object; the text merging operation is used to merge the position relationship description text, object attribute description text and instance name corresponding to the any first instance object into the historical position relationship description text, historical object attribute description text and historical instance name corresponding to the first target instance object, respectively.

[0011] In an embodiment of the present application, the local point cloud map also includes historical semantic feature vectors corresponding to multiple second instance objects, and the text merging operation is also used to merge the semantic feature vector corresponding to any first instance object into the historical semantic feature vector corresponding to the first target instance object; for any first instance object, the point cloud merging operation and the text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map, including: inputting the instance area and instance name of each first instance object into the multimodal semantic extraction model to obtain the semantic feature vectors corresponding to each first instance object; for any first instance object, searching for a first target instance object matching the any first instance object among the multiple second instance objects included in the global point cloud map, and merging the semantic feature vector corresponding to the any first instance object into the historical semantic feature vector corresponding to the first target instance object.

[0012] In an embodiment of the present application, for any first instance object, a point cloud merging operation and a text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map, including: for any first instance object, searching a common instance name that matches the instance name of the any first instance object in a common noun table; for any first instance object, merging the common instance name of the any first instance object into the historical instance name corresponding to the first target instance object.

[0013] In an embodiment of the present application, for any first instance object, performing a point cloud merging operation and a text merging operation on the any first instance object and the first target instance object includes: for any first instance object, adding the position relationship description text and the object attribute description text of the any first instance object to the historical position relationship description text and the historical object attribute description text corresponding to the first target instance object, respectively.

[0014] In an embodiment of the present application, for any first instance object, searching for a first target instance object that matches any first instance object among multiple second instance objects included in a global point cloud map includes: for any first instance object, determining a spatial similarity and a first visual similarity between any second instance object included in the global point cloud map and the any first instance object; for any first instance object, determining a target similarity between any second instance object and the any first instance object based on the spatial similarity and the first visual similarity between any second instance object and any first instance object; for any first instance object, determining a second instance object among multiple second instance objects included in the global point cloud map whose target similarity with the any first instance object is greater than a first preset threshold as a first target instance object that matches the any first instance object.

[0015] In an embodiment of the present application, the method also includes: after the global point cloud map is updated, for any second instance object, if the first detection time of the historical position relationship description text corresponding to any second instance object is before the preset interval frame number, and the number of detections of the historical position relationship description text corresponding to any second instance object is less than the preset number, the historical position relationship description text corresponding to any second instance object is deleted from the updated global point cloud map.

[0016] In an embodiment of the present application, a local point cloud map for a target site corresponding to a current frame is obtained based on a preset algorithm, and the local point cloud map includes first point clouds corresponding to multiple first instance objects, including: obtaining a depth map corresponding to the current frame, and a robot posture when the robot collects the current frame; generating a color depth map corresponding to the current frame according to the current frame and the depth map; generating a local point cloud map for the target site corresponding to the current frame according to the robot posture and the color depth map, and generating a first point cloud and bounding box corresponding to any first instance object in the local point cloud map.

[0017] In an embodiment of the present application, the method also includes: obtaining retrieval information input by a user; loading an updated global point cloud map; determining a target retrieval model corresponding to the retrieval information; inputting the retrieval information into the target retrieval model to output retrieval features corresponding to the retrieval information through the target retrieval model; matching the retrieval features with any second instance object included in the updated global point cloud map to determine a second target instance object that matches the retrieval features in the updated global point cloud map.

[0018] In an embodiment of the present application, determining the target retrieval model corresponding to the retrieval information includes: when the number of words included in the retrieval information is greater than a preset number, determining that the target retrieval model corresponding to the retrieval information is a multimodal semantic extraction model and a long text semantic extraction model; when the number of words included in the retrieval information is less than or equal to a preset number, determining that the target retrieval model corresponding to the retrieval information is a multimodal semantic extraction model and a bag-of-words model.

[0019] In an embodiment of the present application, the retrieval feature includes at least one of a query feature vector, a text feature and a word feature, and the retrieval information is input into the target retrieval model to output the retrieval feature corresponding to the retrieval information through the target retrieval model, including: when the target retrieval model includes a multimodal semantic extraction model, the retrieval information is respectively input into the multimodal semantic extraction model to output the query feature vector corresponding to the retrieval information through the multimodal semantic extraction model; when the target retrieval model includes a long text semantic extraction model, the retrieval information is respectively input into the long text semantic extraction model to output the text feature corresponding to the retrieval information through the long text semantic extraction model; when the target retrieval model includes a bag of words model, the retrieval information is respectively input into the bag of words model to output the word feature corresponding to the retrieval information through the bag of words model.

[0020] In an embodiment of the present application, the retrieval feature includes a query feature vector and a text feature, and matching is performed based on the retrieval feature and any second instance object included in the updated global point cloud map to determine the second target instance object in the updated global point cloud map that matches the retrieval feature, including: determining the second visual similarity between the query feature vector and any second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is greater than a preset number, determining the text similarity between the text feature and each second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is greater than a preset number, determining the second target instance object in the updated global point cloud map that matches the retrieval feature based on the second visual similarity and text similarity corresponding to any second instance object.

[0021] In an embodiment of the present application, the retrieval feature also includes a word feature, and matching is performed based on the retrieval feature and any second instance object included in the updated global point cloud map to determine the second target instance object in the updated global point cloud map that matches the retrieval feature, including: when the number of words included in the retrieval information is less than or equal to a preset number, determining the word similarity between the word feature and each second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is less than or equal to a preset number, determining the second target instance object in the updated global point cloud map that matches the retrieval feature based on the second visual similarity and word similarity corresponding to any second instance object.

[0022] A second aspect of the present application provides an image processing device based on an open vocabulary, comprising:

[0023] a memory configured to store instructions;

[0024] The processor is configured to call instructions from the memory and implement the above-mentioned image processing method based on open vocabulary when executing the instructions.

[0025] A third aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned open vocabulary-based image processing method.

[0026] Through the above technical solution, the first instance object in the current frame is divided by the instance segmentation model, and the current frame is converted into a local point cloud map. The first target instance object matching the first instance object in the local point cloud map is searched in the global point cloud map to perform a point cloud merging operation, which can reduce the ghosting problem in the three-dimensional mapping process and improve the mapping accuracy. In addition, the position relationship description text and object attribute description text of each first instance object are identified through a large language model, and a text merging operation is performed on the first target instance object matching the first instance object, parsing the environment into a semantically rich and structured three-dimensional scene graph, so that the generated three-dimensional scene graph provides a structured and comprehensive understanding of the scene.

[0027] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0029] Figure 1 A schematic diagram of a process flow of an open vocabulary-based image processing method according to an embodiment of the present application is shown;

[0030] Figure 2 A schematic diagram of a process of querying an instance object of a global point cloud image according to an embodiment of the present application is schematically shown;

[0031] Figure 3 The structure block diagram of an open vocabulary-based image processing device according to an embodiment of the present application is schematically shown;

[0032] Figure 4 The schematic diagram shows a structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0034] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0035] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0036] Figure 1 The following schematically shows a flow chart of an open vocabulary-based image processing method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides an open vocabulary-based image processing method, which may include the following steps.

[0037] S102, obtaining a current frame collected by the robot, and inputting the current frame into an instance segmentation model to obtain an instance region and an instance name corresponding to each first instance object.

[0038] S104, obtaining a local point cloud map of the target site corresponding to the current frame based on a preset algorithm, where the local point cloud map includes first point clouds corresponding to a plurality of first instance objects.

[0039] It can be understood that a robot refers to a robot that is movable and can collect images. The current frame refers to the current image corresponding to the local space area collected by the robot when it collects images for the target site at the current moment. The target site refers to the three-dimensional space site specified by the technician for three-dimensional mapping. The instance segmentation model is a technology that combines object detection and semantic segmentation, which is used to identify each independent instance object in the image. ‌Instance segmentation not only needs to distinguish different instance objects in the image, but also needs to generate a mask for each instance object to accurately mark the boundary of each instance object, and can also distinguish the category of the instance object. The instance object refers to an object or area of ​​the instantiation class in the image, such as a table, a chair, a water cup, etc. The first instance object refers to an object or area of ​​the instantiation class in the current frame. Therefore, the current frame collected by the robot is input into the instance segmentation model, and the instance objects in the current frame, as well as the instance area and instance name corresponding to each instance object can be output. It can be understood that the instance area can be characterized by the boundary of the instance object in the current frame, and the instance name refers to the category of the instance object. For example, the instance segmentation model can be a yolo-8l model.

[0040] The local point cloud map refers to the point cloud image of the local space corresponding to the current frame, and the local point cloud map contains the three-dimensional point cloud data of the local space area. Among them, the points in the three-dimensional point cloud data contained in the local point cloud map can belong to different first instance objects. Since the instance segmentation algorithm segments the instance area corresponding to the first instance object, then, in the local point cloud map corresponding to the current frame, according to the position of the instance area in the local point cloud map, the multiple coordinate points in each instance area can be determined as the first point cloud corresponding to the first instance object. The preset algorithm may refer to an image mapping algorithm for converting two-dimensional into three-dimensional. Specifically, it may be an affine transformation matrix. Specifically, in the process of three-dimensional mapping of the target site, after the robot collects the image of the current frame, it can be converted into three-dimensional point cloud data (local point cloud map) in the space area corresponding to the current frame based on the mapping algorithm of converting two-dimensional into three-dimensional.

[0041] In an embodiment of the present application, a local point cloud map for a target site corresponding to a current frame is obtained based on a preset algorithm, and the local point cloud map includes first point clouds corresponding to multiple first instance objects, including: obtaining a depth map corresponding to the current frame, and a robot posture when the robot collects the current frame; generating a color depth map corresponding to the current frame according to the current frame and the depth map; generating a local point cloud map for the target site corresponding to the current frame according to the robot posture and the color depth map, and generating a first point cloud and bounding box corresponding to any first instance object in the local point cloud map.

[0042] It can be understood that the depth map refers to an image with depth information of the current frame. The robot pose can specifically refer to the world coordinates of the camera installed on the robot. ‌A color depth map is an image that combines color information and depth information. ‌It usually consists of three channels, RGB, which record the color information of each pixel, and a separate channel records the distance or depth information between each pixel and the object. A bounding box refers to a rectangle that surrounds each first instance object. Specifically, in the process of three-dimensional mapping of the target site, the robot can continuously collect multiple frames of images, which can be images containing depth information, such as RGB-D images. Each frame of the RGB-D image is combined with the robot's camera pose through a target detection algorithm or a target segmentation algorithm, and then based on a two-dimensional to three-dimensional mapping algorithm, it can be converted into three-dimensional point cloud data (local point cloud map) in the spatial area corresponding to the RGB-D image, and the global three-dimensional point cloud map is constructed and updated through the three-dimensional point cloud data of continuous multiple frames of RGB-D images.

[0043] S106: input all instance regions and all instance names into the first language model and the second language model respectively, so as to obtain a position relationship description text and an object attribute description text for determining each first instance object respectively.

[0044] It can be understood that a large language model (LLM) is an artificial intelligence model designed to understand and generate human language. It usually contains tens of billions (or more) parameters, and obtains a deep understanding of language by training on massive text data. Position relationship description text refers to text that describes the positional relationship between the first instance object and other first instance objects in its local point cloud map through natural language. Object attribute description text refers to text that describes the characteristics and attributes of the first instance object itself through natural language. Since the instance segmentation model obtains the image mask and instance name corresponding to the current frame after segmenting the current frame, the instance area is a partial image mask of the current frame. Therefore, all image masks and instance names can be input into the large language model for recognition by the large language model. Specifically, the first large language model and the second large language model can both be chatGPT4o models. By setting different training data and training parameters, the first large language model can output position relationship description text, while the second large language model can output object attribute description text.

[0045] S108, obtaining a global point cloud map for the target site, the global point cloud map including second point clouds corresponding to multiple second instance objects, historical position relationship description text, historical object attribute description text and historical instance names.

[0046] It can be understood that the global point cloud map refers to the point cloud image of the spatial area corresponding to the entire target site. It can be understood that the objects in the target site may move and change, so the local point cloud map corresponding to the current frame can update the global point cloud map. The global point cloud map includes multiple second instance objects, and the second instance object refers to the object or area of ​​the instantiated class in the global point cloud map, and each second instance object has a corresponding second point cloud. Similarly, the global point cloud map is generated by three-dimensional mapping of the local point cloud maps obtained at multiple historical times, so the second instance objects included in the global point cloud map also have corresponding historical position relationship description texts, historical object attribute description texts and historical instance names. It can be understood that the first instance object is a new instance object at the current moment, and the second instance object is an old instance object at the historical time. Since the first instance object and the second instance object have corresponding attributes respectively, and can be located at their corresponding spatial positions respectively, the global point cloud map can be updated by comparing and analyzing the first instance object and the second instance object.

[0047] S110 , for any first instance object, searching for a first target instance object matching any first instance object among a plurality of second instance objects included in the global point cloud image.

[0048] It can be understood that there is a first instance object in the local point cloud image that is the same as the second instance object in the global point cloud image. When the global point cloud image is updated using the point cloud data of the local point cloud image, there will be duplicate instance objects in the global point cloud image. Then, the second instance object that matches the first instance object in the local point cloud image, that is, the first target instance object, can be found in the global point cloud image. The matching method with the first instance object includes calculating the spatial similarity, visual feature similarity, etc. between the two instance objects.

[0049] In an embodiment of the present application, for any first instance object, searching for a first target instance object that matches any first instance object among multiple second instance objects included in a global point cloud map includes: for any first instance object, determining a spatial similarity and a first visual similarity between any second instance object included in the global point cloud map and the any first instance object; for any first instance object, determining a target similarity between any second instance object and the any first instance object based on the spatial similarity and the first visual similarity between any second instance object and any first instance object; for any first instance object, determining a second instance object among multiple second instance objects included in the global point cloud map whose target similarity with the any first instance object is greater than a first preset threshold as a first target instance object that matches the any first instance object.

[0050] It can be understood that spatial similarity is also called spatial overlap, which can be represented by spatial IoU (Intersection over Union), which is an indicator to measure the overlap of two bounding boxes. The value range of spatial IoU is between 0 and 1, and the higher the value, the higher the overlap of the two bounding boxes. Visual feature similarity refers to the measure of the proximity of two images or objects in visual features. This concept is based on low-level features of images, such as color, texture, shape, etc. Specifically, the calculation of visual similarity usually involves core elements such as global similarity, local similarity, and structural similarity. Among them, global similarity considers features such as color, texture, and shape of the entire image. Local similarity focuses on the features of specific areas in the image, such as the details of an object or scene. Structural similarity can measure the structural relationship between images, such as the layout of objects and scenes. For any first instance object, the spatial similarity between the first instance object and each second instance object can be determined by calculating the spatial IoU through the bounding box corresponding to the first instance object and the bounding box corresponding to each second instance object. Furthermore, for any first instance object, the first visual similarity between the two can be calculated by the feature vector representing the visual features of any first instance object and any second instance object. Specifically, the feature vector representing the visual features can be a semantic feature vector. Further, the spatial similarity and the first visual similarity are weighted and summed to obtain the final target similarity between any first instance object and any second instance object. The size of the target similarity is used to determine whether any second instance object is a first target instance object that matches any first instance object. Here, a first preset threshold can be set as a judgment condition. The first preset threshold refers to the experience value set by the technician for the target similarity based on technical experience. If there is a second instance object in the global point cloud map whose target similarity between the first instance object is greater than the first preset threshold, the second instance object can be selected as the first target instance object. That is, when the weighted sum of the spatial similarity and the first visual similarity meets the conditions set by the technician, the second instance object in the global point cloud map can be considered to be the first target instance object that matches the first instance object. ‌‌

[0051] S112, for any first instance object, perform a point cloud merging operation and a text merging operation on the any first instance object and the first target instance object to update the global point cloud map; wherein the point cloud merging operation is used to merge the first point cloud corresponding to the any first instance object into the second point cloud corresponding to the first target instance object; the text merging operation is used to merge the position relationship description text, object attribute description text and instance name corresponding to the any first instance object into the historical position relationship description text, historical object attribute description text and historical instance name corresponding to the first target instance object, respectively.

[0052] It can be understood that the first point cloud corresponding to the matched first instance object and the second point cloud corresponding to the target instance object are subjected to a point cloud merging operation. Specifically, the first point cloud contains multiple first coordinate points, and the second point cloud also contains multiple second coordinate points. Among them, each coordinate point has a unique identifier. For the matched first instance object and the target instance object, each first coordinate point can be matched one by one with each second coordinate point for merging, so that the first point cloud corresponding to any first instance object is merged into the second point cloud corresponding to the first target instance object, and the instance object merging operation in the spatial dimension is completed. At the same time, the matched first instance object and the first target instance object can also merge the relevant attributes of the two. Specifically, the position relationship description text, object attribute description text and instance name corresponding to any first instance object can be respectively merged into the historical position relationship description text, historical object attribute description text and historical instance name corresponding to the first target instance object.

[0053] In an embodiment of the present application, for any first instance object, performing a point cloud merging operation and a text merging operation on the any first instance object and the first target instance object includes: for any first instance object, adding the position relationship description text and the object attribute description text of the any first instance object to the historical position relationship description text and the historical object attribute description text corresponding to the first target instance object, respectively.

[0054] Specifically, the position relationship description text and the object attribute description text can be directly added to the corresponding historical position relationship description text and the historical object attribute description text. Or the position relationship description text and the object attribute description text directly replace the corresponding historical position relationship description text and the historical object attribute description text. The merging of the instance name and the historical instance name can be the number of merging times corresponding to the instance name plus 1. In a specific implementation, for other first instance objects that do not match the first target instance object in the global point cloud map, their corresponding first point clouds and the corresponding position relationship description text, object attribute description text and instance name are directly added to the global point cloud map.

[0055] Through the above technical solution, the first instance object in the current frame is divided by the instance segmentation model, and the current frame is converted into a local point cloud map. The first target instance object matching the first instance object in the local point cloud map is searched in the global point cloud map to perform a point cloud merging operation, which can reduce the ghosting problem in the three-dimensional mapping process and improve the mapping accuracy. In addition, the position relationship description text and object attribute description text of each first instance object are identified by a large language model, and the text merging operation is performed on the first target instance object matching the first instance object, so that the visual image can achieve multi-scale fusion and have semantic information, and the environment is parsed into a semantically rich and structured three-dimensional scene graph, so that the generated three-dimensional scene graph provides a structured and comprehensive understanding of the scene.

[0056] In an embodiment of the present application, the global point cloud map also includes historical semantic feature vectors corresponding to multiple second instance objects, and the text merging operation is also used to merge the semantic feature vector corresponding to any first instance object into the historical semantic feature vector corresponding to the first target instance object; for any first instance object, the point cloud merging operation and the text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map, including: inputting the instance area and instance name of each first instance object into the multimodal semantic extraction model to obtain the semantic feature vectors corresponding to each first instance object; for any first instance object, searching for a first target instance object matching the any first instance object among the multiple second instance objects included in the global point cloud map, and merging the semantic feature vector corresponding to the any first instance object into the historical semantic feature vector corresponding to the first target instance object.

[0057] It can be understood that a semantic feature vector is a feature vector used to represent the meaning in text or images, which captures the semantic information of the data. In computer vision, semantic features may be representations of objects, scenes, or attributes in an image. These features can be extracted by convolutional neural networks (CNNs) or pre-trained image classification models such as ResNet, Inception, etc. These models are trained to recognize different objects and scenes in images and represent them in vector form for further processing and analysis. Specifically, a multimodal semantic extraction model can be used to extract the semantic feature vector corresponding to each first instance object in the current frame. Among them, the multimodal semantic extraction model is an artificial intelligence model that can process different types of data. It integrates multiple input information such as vision, language, and audio to achieve more comprehensive and accurate understanding and reasoning capabilities. This model can capture richer semantic and contextual information by correlating data from different modalities, thereby providing more in-depth analysis and prediction. Traditional unimodal models can only process a certain type of data, such as image classification or text generation, but problems in actual processes often require the combination of multiple information sources. The multimodal semantic model can better understand and reason about complex information by fusing data from multiple modalities, thereby providing more intelligent solutions. Then, the corresponding second instance object in the global point cloud map also has a corresponding historical semantic feature vector. Then, for any first instance object, a first target instance object matching the any first instance object is searched among the multiple second instance objects included in the global point cloud map, and the semantic feature vector corresponding to the any first instance object is merged into the historical semantic feature vector corresponding to the first target instance object. For example, the multimodal semantic extraction model can be a ‌BGE-visualized-m3 model, and the semantic feature vector can be a fused 1024-dimensional feature vector. ‌

[0058] In an embodiment of the present application, for any first instance object, a point cloud merging operation and a text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map, including: for any first instance object, searching a common instance name that matches the instance name of the any first instance object in a common noun table; for any first instance object, merging the common instance name of the any first instance object into the historical instance name corresponding to the first target instance object.

[0059] It can be understood that instance objects with different instance names may actually be the same. For example, a laptop and a notebook computer. Alternatively, instance objects with different instance names may belong to the same concept category, for example, a thermos cup and a mug both belong to the category of water cups. In order to facilitate the search and analysis of instance objects in an image, a common ranking table can be set. Search the common instance name that matches the instance name of the first instance object in the common ranking table, and replace the instance name with the corresponding common instance name. It can be considered that the common instance name is the name of the superordinate concept corresponding to the instance name. Then, for any first instance object, the common instance name of any first instance object is merged into the historical instance name corresponding to the first target instance object.

[0060] In an embodiment of the present application, the method also includes: after the global point cloud map is updated, for any second instance object, if the first detection time of the historical position relationship description text corresponding to any second instance object is before the preset interval frame number, and the number of detections of the historical position relationship description text corresponding to any second instance object is less than the preset number, the historical position relationship description text corresponding to any second instance object is deleted from the updated global point cloud map.

[0061] It can be understood that the preset interval frame number is the interval frame number preset by the technician. The preset number of detections is the number of detections preset by the technician. For some inactive instance objects, such instance objects appear very rarely, which may be due to measurement errors. Therefore, it is necessary to regularly clean up the location information description text of inactive instance objects. For any second instance object, if the first detection time of the historical position relationship description text corresponding to any second instance object is before the preset interval frame number, and the number of detections of the historical position relationship description text corresponding to any second instance object is less than the preset number, the historical position relationship description text corresponding to any second instance object will be deleted from the updated global point cloud map. For example, the preset interval frame number can be 5 frames, and the preset number of times can be 2 times.

[0062] In one embodiment, the point cloud merging operation and the text merging operation are repeated for each RGB-D image (continuous frame), and the cycle is repeated. However, after an interval of m images, the current global point cloud map can be subjected to a denoising, filtering, and instance object merging operation. Denoising and filtering refer to the input of the current global point cloud map. For the last RGB-D image, the global point cloud map is additionally subjected to denoising, filtering, and instance object merging operations to obtain the final global point cloud map information.

[0063] In an embodiment of the present application, the method also includes: obtaining retrieval information input by a user; loading an updated global point cloud map; determining a target retrieval model corresponding to the retrieval information; inputting the retrieval information into the target retrieval model to output retrieval features corresponding to the retrieval information through the target retrieval model; matching the retrieval features with any second instance object included in the updated global point cloud map to determine a second target instance object that matches the retrieval features in the updated global point cloud map.

[0064] It can be understood that after the global point cloud map is updated, each second instance object included in the updated global point cloud map has corresponding text description information. The user can query and search through the global point cloud map. It can be understood that the user can search for one global point cloud map or multiple global point cloud maps. The retrieval information input by the user can be a piece of text, and the updated global point cloud map can be loaded according to the retrieval information. According to the retrieval information, the corresponding target retrieval model can be determined. There can be multiple retrieval models, including but not limited to various semantic models such as multimodal semantic models, bag of words models, and long text semantic extraction models. The retrieval information input by the user is analyzed and processed by the target retrieval model, and the retrieval features corresponding to the retrieval information are output. According to the retrieval features, the updated global point cloud map is searched, and the second instance object matching the retrieval features is used as the second target instance object, and the second target instance object is the query result.

[0065] In an embodiment of the present application, determining the target retrieval model corresponding to the retrieval information includes: when the number of words included in the retrieval information is greater than a preset number, determining that the target retrieval model corresponding to the retrieval information is a multimodal semantic extraction model and a long text semantic extraction model; when the number of words included in the retrieval information is less than or equal to a preset number, determining that the target retrieval model corresponding to the retrieval information is a multimodal semantic extraction model and a bag-of-words model.

[0066] In an embodiment of the present application, the retrieval feature includes at least one of a query feature vector, a text feature and a word feature, and the retrieval information is input into the target retrieval model to output the retrieval feature corresponding to the retrieval information through the target retrieval model, including: when the target retrieval model includes a multimodal semantic extraction model, the retrieval information is respectively input into the multimodal semantic extraction model to output the query feature vector corresponding to the retrieval information through the multimodal semantic extraction model; when the target retrieval model includes a long text semantic extraction model, the retrieval information is respectively input into the long text semantic extraction model to output the text feature corresponding to the retrieval information through the long text semantic extraction model; when the target retrieval model includes a bag of words model, the retrieval information is respectively input into the bag of words model to output the word feature corresponding to the retrieval information through the bag of words model.

[0067] As you can understand, the Bag-of-words model is a commonly used text representation method in natural language processing and information retrieval. It represents text as a collection of words, ignoring the order and grammatical structure of words, and only focusing on the frequency of word occurrence. It is a model used to process and analyze long text data, aiming to capture and understand the semantic information in the text. ‌ This model is particularly important in the field of natural language processing (NLP) because it can help machines understand the deep meaning and contextual relationships of complex texts such as long documents and long paragraphs. The core of the long text semantic model is that it can process and analyze the complex semantic features of long texts. This type of model usually includes two main parts: an encoder and a decoder. The encoder is responsible for receiving the input text sequence and converting it into a context vector through a series of neural network layers. This vector contains all the information of the input text and provides a basis for the decoder to generate text. The decoder gradually generates the output text sequence according to the context vector provided by the encoder. At each step, it predicts the next word and adds it to the output sequence. The query feature vector is the semantic feature vector corresponding to the retrieval information identified by the multimodal semantic extraction model. The text feature is the feature corresponding to the retrieval information identified by the long text semantic extraction model. The word feature refers to the feature corresponding to the retrieval information identified by the bag-of-words model. Specifically, the multimodal semantic extraction model can be a ‌BGE-visualized-m3 model, and the text semantic extraction model can be a ‌BGE-m3 model.

[0068] In an embodiment of the present application, the retrieval feature includes a query feature vector and a text feature, and matching is performed based on the retrieval feature and any second instance object included in the updated global point cloud map to determine the second target instance object in the updated global point cloud map that matches the retrieval feature, including: determining the second visual similarity between the query feature vector and any second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is greater than a preset number, determining the text similarity between the text feature and each second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is greater than a preset number, determining the second target instance object in the updated global point cloud map that matches the retrieval feature based on the second visual similarity and text similarity corresponding to any second instance object.

[0069] In an embodiment of the present application, the retrieval feature also includes a word feature, and matching is performed based on the retrieval feature and any second instance object included in the updated global point cloud map to determine the second target instance object in the updated global point cloud map that matches the retrieval feature, including: when the number of words included in the retrieval information is less than or equal to a preset number, determining the word similarity between the word feature and each second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is less than or equal to a preset number, determining the second target instance object in the updated global point cloud map that matches the retrieval feature based on the second visual similarity and word similarity corresponding to any second instance object.

[0070] Specifically, refer to Figure 2, the user inputs the search information (input_query), then loads the saved global point cloud map, and returns the location of the second target instance object corresponding to the search information through the query module. Among them, the ‌BGE-visualized-m3 model inputs input_query to obtain the query feature vector, and calculates the cosine similarity between the query feature vector and the clip_ft of each instance object in the global point cloud map to obtain the second visual similarity. Generally speaking, although the model inputs text, since the training of ‌BGE-visualized-m3 is text-image pairs. Then, it can be considered that the ‌BGE-visualized-m3 model can match image features. The ‌BGE-m3 model inputs the information description text caption (the information description text includes the position relationship description text and the object attribute description text) and input_query of each second instance object in the global point cloud map, obtains the text features corresponding to each instance object and the text features of input_query, and calculates the cosine similarity between the text features of input_query and the text features corresponding to each second instance object as the text similarity. The ‌BGE-m3 model is a pure text model, and its training data set is text. It has good support for long texts, so the ‌BGE-m3 model is used to understand long texts. Then, it can be considered that the text features can be matched through the ‌BGE-m3 model, so the long sentence query adopts the ‌BGE-m3 model and image feature fusion. The bag-of-words model inputs the instance name class_name and input_query of each second instance object in the global point cloud map, obtains the word features corresponding to each second instance object and the word features of input_query, and calculates the cosine similarity between the word features of input_query and the word features corresponding to each second instance object as the word similarity. The bag-of-words model has high accuracy for word phrase matching, so the short word query adopts the bag-of-words model and image feature fusion. Therefore, after entering the search information (input_query), it is necessary to determine whether the search information is greater than the preset number of words. The preset number is the number of words preset by the technician. Specifically, the preset number can be set to 2 words. If it is greater than 2 words, the ‌BGE-visualized-m3 model combined with the ‌BGE-m3 model is used for recognition. If it is less than or equal to 2 words, the ‌BGE-visualized-m3 model combined with the bag-of-words model is used for recognition. Furthermore, for each instance object, the comprehensive similarity corresponding to the input_query is obtained.Specifically, the comprehensive similarity is the weighted sum of the second visual similarity, text similarity (when greater than 2 words) or the second visual similarity, word similarity (when less than or equal to 2 words) between the input_query and each second instance object, and the second instance object with the highest comprehensive similarity is selected as the result of the user query.

[0071] Through the above technical solution, the first instance object in the current frame is divided by the instance segmentation model, and the current frame is converted into a local point cloud map. The first target instance object matching the first instance object of the local point cloud map is searched in the global point cloud map to perform a point cloud merging operation, which can reduce the ghosting problem in the three-dimensional mapping process. In addition, the position relationship description text and object attribute description text of each first instance object are identified by a large language model, and the text merging operation is performed on the first target instance object matching the first instance object, so that the visual image can achieve multi-scale fusion and have semantic information, so that the generated three-dimensional scene graph provides a structured and comprehensive understanding of the scene. Further, by adopting a combined retrieval method, multi-scale fusion retrieval can be achieved. Compared with the query accuracy of 70%-80% of the prior art, the above solution can improve the query accuracy by about 15% on this basis. In addition, this technology is applied in the fields of industrial automation, smart home, autonomous driving, virtual reality and augmented reality, so that the device can perform efficient perception and planning based on natural language instructions, and can perform tasks by understanding the instructions of natural language of open vocabulary.

[0072] Figure 1 FIG. 1 is a flow chart of an open vocabulary-based image processing method in one embodiment. Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0073] Figure 3 The structure block diagram of a controller according to an embodiment of the present application is schematically shown. Figure 3 As shown, the embodiment of the present application provides an image processing device based on an open vocabulary, which may include:

[0074] a memory configured to store instructions;

[0075] The processor is configured to call instructions from the memory and implement the above-mentioned image processing device based on open vocabulary when executing the instructions.

[0076] Specifically, in the embodiment of the present application, the processor may be configured to:

[0077] Obtaining a current frame collected by the robot, and inputting the current frame into an instance segmentation model to obtain an instance region and an instance name corresponding to each first instance object;

[0078] Acquire a local point cloud map corresponding to the target site in the current frame based on a preset algorithm, where the local point cloud map includes first point clouds corresponding to a plurality of first instance objects;

[0079] Inputting all instance regions and all instance names into the first language model and the second language model respectively, so as to obtain a position relationship description text and an object attribute description text for determining each first instance object respectively;

[0080] Acquire a global point cloud map for the target site, the global point cloud map including second point clouds corresponding to multiple second instance objects, historical position relationship description text, historical object attribute description text, and historical instance names;

[0081] For any first instance object, searching for a first target instance object matching the first instance object among a plurality of second instance objects included in the global point cloud image;

[0082] For any first instance object, a point cloud merging operation and a text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map; wherein the point cloud merging operation is used to merge the first point cloud corresponding to the any first instance object into the second point cloud corresponding to the first target instance object; the text merging operation is used to merge the position relationship description text, object attribute description text and instance name corresponding to the any first instance object into the historical position relationship description text, historical object attribute description text and historical instance name corresponding to the first target instance object, respectively.

[0083] In the embodiment of the present application, the processor may be configured to:

[0084] The local point cloud map also includes historical semantic feature vectors corresponding to multiple second instance objects, and the text merging operation is also used to merge the semantic feature vector corresponding to any first instance object into the historical semantic feature vector corresponding to the first target instance object; for any first instance object, the point cloud merging operation and the text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map, including: inputting the instance area and instance name of each first instance object into the multimodal semantic extraction model to obtain the semantic feature vectors corresponding to each first instance object; for any first instance object, searching for a first target instance object matching the any first instance object among the multiple second instance objects included in the global point cloud map, and merging the semantic feature vector corresponding to the any first instance object into the historical semantic feature vector corresponding to the first target instance object.

[0085] In the embodiment of the present application, the processor may be configured to:

[0086] For any first instance object, search the common noun table for a common instance name that matches the instance name of the first instance object; for any first instance object, merge the common instance name of the first instance object into the historical instance name corresponding to the first target instance object.

[0087] In the embodiment of the present application, the processor may be configured to:

[0088] For any first instance object, the position relationship description text and object attribute description text of any first instance object are respectively added to the historical position relationship description text and historical object attribute description text corresponding to the first target instance object.

[0089] In the embodiment of the present application, the processor may be configured to:

[0090] For any first instance object, determine the spatial similarity and the first visual similarity between any second instance object included in the global point cloud image and any first instance object; for any first instance object, determine the target similarity between any second instance object and any first instance object based on the spatial similarity and the first visual similarity between any second instance object and any first instance object; for any first instance object, determine a second instance object among multiple second instance objects included in the global point cloud image, whose target similarity with any first instance object is greater than a first preset threshold, as a first target instance object matching the any first instance object.

[0091] In the embodiment of the present application, the processor may be configured to:

[0092] After the global point cloud map is updated, for any second instance object, if the first detection time of the historical position relationship description text corresponding to any second instance object is before the preset interval frame number, and the number of detections of the historical position relationship description text corresponding to any second instance object is less than the preset number, the historical position relationship description text corresponding to any second instance object will be deleted from the updated global point cloud map.

[0093] In the embodiment of the present application, the processor may be configured to:

[0094] Obtain a depth map corresponding to the current frame and the robot posture when the robot collects the current frame; generate a color depth map corresponding to the current frame based on the current frame and the depth map; generate a local point cloud map for the target site corresponding to the current frame based on the robot posture and the color depth map, and generate a first point cloud and bounding box corresponding to any first instance object in the local point cloud map.

[0095] In the embodiment of the present application, the processor may be configured to:

[0096] Obtain retrieval information input by the user; load the updated global point cloud map; determine the target retrieval model corresponding to the retrieval information; input the retrieval information into the target retrieval model to output retrieval features corresponding to the retrieval information through the target retrieval model; match the retrieval features with any second instance object included in the updated global point cloud map to determine the second target instance object matching the retrieval features in the updated global point cloud map.

[0097] In the embodiment of the present application, the processor may be configured to:

[0098] When the number of words included in the search information is greater than a preset number, the target retrieval model corresponding to the search information is determined to be a multimodal semantic extraction model and a long text semantic extraction model; when the number of words included in the search information is less than or equal to a preset number, the target retrieval model corresponding to the search information is determined to be a multimodal semantic extraction model and a bag-of-words model.

[0099] In the embodiment of the present application, the processor may be configured to:

[0100] The retrieval features include at least one of a query feature vector, a text feature and a word feature. The retrieval information is input into a target retrieval model to output the retrieval features corresponding to the retrieval information through the target retrieval model. The steps include: when the target retrieval model includes a multimodal semantic extraction model, the retrieval information is respectively input into the multimodal semantic extraction model to output the query feature vector corresponding to the retrieval information through the multimodal semantic extraction model; when the target retrieval model includes a long text semantic extraction model, the retrieval information is respectively input into the long text semantic extraction model to output the text features corresponding to the retrieval information through the long text semantic extraction model; when the target retrieval model includes a bag of words model, the retrieval information is respectively input into the bag of words model to output the word features corresponding to the retrieval information through the bag of words model.

[0101] In the embodiment of the present application, the processor may be configured to:

[0102] The retrieval features include a query feature vector and a text feature. Matching is performed based on the retrieval features and any second instance object included in the updated global point cloud map to determine the second target instance object in the updated global point cloud map that matches the retrieval features, including: determining the second visual similarity between the query feature vector and any second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is greater than a preset number, determining the text similarity between the text feature and each second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is greater than a preset number, determining the second target instance object in the updated global point cloud map that matches the retrieval features based on the second visual similarity and text similarity corresponding to any second instance object.

[0103] In the embodiment of the present application, the processor may be configured to:

[0104] The retrieval feature also includes a word feature, and matching is performed based on the retrieval feature and any second instance object included in the updated global point cloud map to determine the second target instance object in the updated global point cloud map that matches the retrieval feature, including: when the number of words included in the retrieval information is less than or equal to a preset number, determining the word similarity between the word feature and each second instance object included in the updated global point cloud map; when the number of words included in the retrieval information is less than or equal to a preset number, determining the second target instance object in the updated global point cloud map that matches the retrieval feature based on the second visual similarity and word similarity corresponding to any second instance object.

[0105] An embodiment of the present application further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned open vocabulary-based image processing method.

[0106] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store image processing data based on an open vocabulary. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, an image processing method based on an open vocabulary is implemented.

[0107] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An image processing method based on open vocabulary, characterized in that: The method comprises: Acquire a current frame collected by the robot, and input the current frame into an instance segmentation model to obtain an instance region and an instance name corresponding to each first instance object; Acquire a local point cloud map corresponding to the current frame and corresponding to the target site based on a preset algorithm, wherein the local point cloud map includes first point clouds corresponding to a plurality of first instance objects; Inputting all instance regions and all instance names into the first language model and the second language model respectively, so as to obtain a position relationship description text and an object attribute description text for determining each first instance object respectively; Acquire a global point cloud map for the target site, the global point cloud map including second point clouds corresponding to a plurality of second instance objects, historical position relationship description text, historical object attribute description text, and historical instance names; For any first instance object, searching for a first target instance object matching the first instance object among a plurality of second instance objects included in the global point cloud image; For any first instance object, a point cloud merging operation and a text merging operation are performed on the any first instance object and the first target instance object to update the global point cloud map; wherein the point cloud merging operation is used to merge the first point cloud corresponding to the any first instance object into the second point cloud corresponding to the first target instance object; the text merging operation is used to merge the position relationship description text, object attribute description text and instance name corresponding to the any first instance object into the historical position relationship description text, historical object attribute description text and historical instance name corresponding to the first target instance object, respectively.

2. The open vocabulary based image processing method according to claim 1, characterized in that: The global point cloud map also includes historical semantic feature vectors corresponding to a plurality of second instance objects, and the text merging operation is further used to merge the semantic feature vector corresponding to any first instance object into the historical semantic feature vector corresponding to the first target instance object; The step of performing a point cloud merging operation and a text merging operation on any first instance object and the first target instance object to update the global point cloud map includes: Inputting the instance region and the instance name of each first instance object into the multimodal semantic extraction model to obtain a semantic feature vector corresponding to each first instance object; For any first instance object, a first target instance object matching the any first instance object is searched among multiple second instance objects included in the global point cloud image, and the semantic feature vector corresponding to the any first instance object is merged into the historical semantic feature vector corresponding to the first target instance object.

3. The open vocabulary based image processing method according to claim 1, characterized in that: The step of performing a point cloud merging operation and a text merging operation on any first instance object and the first target instance object to update the global point cloud map includes: For any first instance object, searching the common noun table for a common instance name that matches the instance name of the first instance object; For any first instance object, the common instance name of the first instance object is merged into the historical instance name corresponding to the first target instance object.

4. The open vocabulary based image processing method according to claim 1, characterized in that: The performing of the point cloud merging operation and the text merging operation on any first instance object and the first target instance object comprises: For any first instance object, the position relationship description text and the object attribute description text of any first instance object are respectively added to the historical position relationship description text and the historical object attribute description text corresponding to the first target instance object.

5. The open vocabulary based image processing method according to claim 1, characterized in that: The step of searching, for any first instance object, a first target instance object matching any first instance object among a plurality of second instance objects included in the global point cloud image comprises: For any first instance object, determine a spatial similarity and a first visual similarity between any second instance object included in the global point cloud image and the any first instance object; For any first instance object, determining a target similarity between any second instance object and any first instance object according to a spatial similarity and a first visual similarity between any second instance object and any first instance object; For any first instance object, a second instance object whose target similarity with any first instance object is greater than a first preset threshold among multiple second instance objects included in the global point cloud image is determined as a first target instance object matching the any first instance object.

6. The open vocabulary based image processing method according to claim 1, characterized in that: The method further comprises: After the updated global point cloud map, for any second instance object, if the first detection time of the historical position relationship description text corresponding to any second instance object is before the preset interval frame number, and the number of detections of the historical position relationship description text corresponding to any second instance object is less than the preset number, the historical position relationship description text corresponding to any second instance object will be deleted from the updated global point cloud map.

7. The open vocabulary based image processing method according to claim 1, characterized in that: The obtaining of a local point cloud map corresponding to the target site in the current frame based on a preset algorithm, wherein the local point cloud map includes first point clouds corresponding to a plurality of first instance objects, including: Acquire a depth map corresponding to the current frame and a robot posture when the robot acquires the current frame; Generate a color depth map corresponding to the current frame according to the current frame and the depth map; A local point cloud map for the target site corresponding to the current frame is generated according to the robot posture and the color depth map, and a first point cloud and a bounding box corresponding to any first instance object are generated in the local point cloud map.

8. The open vocabulary based image processing method according to claim 1, characterized in that: The method further comprises: Get the search information entered by the user; Load the updated global point cloud image; Determining a target retrieval model corresponding to the retrieval information; Inputting the search information into the target search model so as to output search features corresponding to the search information through the target search model; Matching is performed according to the retrieval feature and any second instance object included in the updated global point cloud image, and a second target instance object matching the retrieval feature is determined in the updated global point cloud image.

9. The open vocabulary based image processing method according to claim 8, characterized in that: Determining the target retrieval model corresponding to the retrieval information includes: In a case where the number of words included in the search information is greater than a preset number, determining that the target search model corresponding to the search information is a multimodal semantic extraction model and a long text semantic extraction model; When the number of words included in the search information is less than or equal to the preset number, it is determined that the target search model corresponding to the search information is the multimodal semantic extraction model and the bag-of-words model.

10. The open vocabulary based image processing method according to claim 8, characterized in that: The retrieval feature includes at least one of a query feature vector, a text feature, and a word feature, and the step of inputting the retrieval information into the target retrieval model to output the retrieval feature corresponding to the retrieval information through the target retrieval model includes: In the case where the target retrieval model includes a multimodal semantic extraction model, the retrieval information is respectively input into the multimodal semantic extraction model, so as to output a query feature vector corresponding to the retrieval information through the multimodal semantic extraction model; The target retrieval model includes a long text semantic extraction model, and the retrieval information is respectively input into the long text semantic extraction model, so that the text features corresponding to the retrieval information are output through the long text semantic extraction model; In the case where the target retrieval model includes a bag-of-words model, the retrieval information is respectively input into the bag-of-words model, so that the word features corresponding to the retrieval information are output through the bag-of-words model.

11. The open vocabulary based image processing method according to claim 8, characterized in that: The retrieval feature includes a query feature vector and a text feature, and the matching between the retrieval feature and any second instance object included in the updated global point cloud image to determine the second target instance object in the updated global point cloud image that matches the retrieval feature includes: Determining a second visual similarity between the query feature vector and any second instance object included in the updated global point cloud image; In a case where the number of words included in the search information is greater than a preset number, determining a text similarity between the text feature and each second instance object included in the updated global point cloud image; In a case where the number of words included in the search information is greater than a preset number, a second target instance object matching the search feature in the updated global point cloud image is determined according to a second visual similarity and a text similarity corresponding to any second instance object.

12. The open vocabulary based image processing method according to claim 11, characterized in that: The retrieval feature further includes a word feature, and the matching between the retrieval feature and any second instance object included in the updated global point cloud image to determine the second target instance object in the updated global point cloud image that matches the retrieval feature includes: In a case where the number of words included in the search information is less than or equal to the preset number, determining a word similarity between the word feature and each second instance object included in the updated global point cloud image; When the number of words included in the search information is less than or equal to the preset number, a second target instance object matching the search feature in the updated global point cloud image is determined according to the second visual similarity and word similarity corresponding to any second instance object.

13. An image processing device based on open vocabulary, characterized in that: include: a memory configured to store instructions; A processor is configured to call the instructions from the memory and implement the open vocabulary based image processing method according to any one of claims 1 to 12 when executing the instructions.

14. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing a machine to execute the open vocabulary-based image processing method according to any one of claims 1 to 12.

Images