Object position detection method, semantic prediction model training method and electronic equipment

Abstract

The embodiment of the invention is applied to the technical field of computer vision, and provides an object position detection method, a semantic prediction model training method and electronic equipment. The electronic device collects a target look-around image. Wherein the target look-around image is used for representing a three-dimensional scene of an environment where the electronic equipment is located. Afterwards, under the condition that a query instruction is received, the electronic equipment can utilize the target semantic prediction model to carry out semantic prediction on the target around-view image, and semantic information of a three-dimensional scene is obtained. Wherein the semantic information comprises object information corresponding to each voxel in the three-dimensional scene. Afterwards, the electronic device can determine the position information of the to-be-queried object in the three-dimensional scene according to the semantic information of the three-dimensional scene and the object information of the to-be-queried object. According to the invention, accurate detection of the position of the object can be realized.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an object position detection method, a semantic prediction model training method, and an electronic device. Background Technology

[0002] With the continuous development of computer vision technology, 3D semantic prediction methods based on computer vision can be applied to an increasing number of scenarios (such as autonomous driving and robot movement) to provide convenience for users. The task of this 3D semantic prediction method is to determine the location information of the query object in a 3D scene based on the acquired images. Accurately determining the location information of the query object in a 3D scene is fundamental for electronic devices to perceive their surroundings and is crucial for subsequent target operations (such as obstacle avoidance and path planning).

[0003] Therefore, accurately determining the position information of the object to be queried in a 3D scene so that electronic devices can perform subsequent operations is an urgent problem to be solved. Summary of the Invention

[0004] This application provides an object location detection method, a semantic prediction model training method, and an electronic device to achieve accurate object location detection, improve the detection accuracy of the location of the object to be queried, and thus enhance the user experience.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, an object position detection method is provided, applied to an electronic device. In this method, the electronic device acquires a target surround view image. This target surround view image represents a three-dimensional scene of the environment in which the electronic device is located. Then, upon receiving a query command, the electronic device can use a target semantic prediction model to perform semantic prediction on the target surround view image, obtaining semantic information of the three-dimensional scene. This semantic information includes object information corresponding to each voxel in the three-dimensional scene. Subsequently, the electronic device can determine the position information of the queried object in the three-dimensional scene based on the semantic information of the three-dimensional scene and the object information of the queried object.

[0007] In this application, the electronic device obtains object information corresponding to each voxel in the 3D scene to which the target surround view image belongs by performing semantic prediction. This allows the electronic device to determine the position of the target object within its environment. This enables accurate detection of the object's location, facilitating subsequent operations by the electronic device and ultimately improving the user experience.

[0008] In one possible implementation of the first aspect, the query instruction includes attribute information and / or category information representing open terms, and the object information of the object to be queried includes attribute information and / or category information.

[0009] In this application, since object information can include not only attribute information and / or category information representing fixed terms, but also attribute information and / or category information representing open terms, electronic devices can query the three-dimensional position information of corresponding objects for any open term, without being limited to any fixed term in the fixed term list. This enables flexible detection of object positions, reduces the occurrence of situations where the location of the queried object cannot be found due to the query command being an open term, improves the detection accuracy of the queried object, and thus enhances the user experience.

[0010] In one possible implementation of the first aspect, the process of semantic prediction of the target surround view image by the electronic device may specifically include: the electronic device inputting the target surround view image into a target semantic prediction model to obtain three-dimensional raster features. These three-dimensional raster features include raster features of multiple raster cells, which are used to characterize object information of voxels in the three-dimensional scene.

[0011] In this application, the semantic information of a three-dimensional scene is represented by three-dimensional grid features, which enables electronic devices to accurately determine the object information of each voxel in the three-dimensional scene, thereby providing convenient conditions for subsequent accurate detection of object positions.

[0012] In one possible implementation of the first aspect, the process of the electronic device determining location information may specifically include: the electronic device recognizing the object information of the object to be queried to obtain query text. Then, the electronic device inputs the query text into a visual language model to obtain the linguistic features of the object to be queried; wherein the linguistic features are used to characterize the object information of the object to be queried. Next, the electronic device matches the linguistic features of the object to be queried with three-dimensional raster features to obtain the location information of the object to be queried in the three-dimensional scene.

[0013] In this application, after receiving a query command, the electronic device can recognize the query command, obtain the query text, and extract features from the query text to obtain the linguistic features of the target object. This enables accurate extraction of object features from the query command, thus facilitating subsequent accurate detection of the object's location.

[0014] In one possible implementation of the first aspect, the process of matching the linguistic features of the object to be queried with the 3D raster features by the electronic device may specifically include: for each raster feature in the 3D raster features, the electronic device calculates the similarity between the raster features and the linguistic features of the object to be queried. Then, the electronic device uses the raster features with similarity greater than a preset similarity to form a 3D mask of the object to be queried. The 3D mask of the object to be queried is used to represent the positional information of the object to be queried in the 3D scene.

[0015] In this application, the electronic device determines the location information of the target object in the environment of the electronic device by calculating the similarity between features. This can improve the accuracy of real-time object location detection, so that users can quickly find the target object and improve the user experience.

[0016] In one possible implementation of the first aspect, the training process of the aforementioned target semantic prediction model may specifically include: an electronic device acquiring a sample database. This sample database includes at least one sample image showing at least one object. Subsequently, the electronic device can train a pre-constructed semantic prediction model based on the at least one sample image to obtain the target semantic prediction model. The output of this target semantic prediction model is used to characterize the semantic information of the three-dimensional scene.

[0017] In this application, the target semantic prediction model can be trained solely based on sample images without relying on manually labeled semantic tags. This not only saves a significant amount of time and manpower costs, thus improving the training efficiency of the semantic prediction model, but also allows the target semantic prediction model to be unrestricted by manually labeled semantic tags and to perform semantic prediction on any open vocabulary. This enhances the application flexibility and semantic prediction effect of the target semantic prediction model, thereby facilitating the accurate detection of object positions.

[0018] In one possible implementation of the first aspect, the process of training a semantic prediction model by the electronic device may specifically include: for each sample image in the sample database, the electronic device may adjust the parameters of an initialized three-dimensional Gaussian sphere according to the sample image to obtain the target three-dimensional Gaussian sphere features of the sample image. Then, the electronic device may render the target three-dimensional Gaussian sphere features according to a Gaussian splashing model to obtain the predicted language features of the sample image. Subsequently, the electronic device may train a pre-constructed semantic prediction model based on the predicted language features and the true language features of the sample image to obtain the target semantic prediction model; wherein the true language features of the sample image are obtained based on the sample image.

[0019] In this application, a 3D scene model adapted to the target 3D scene is represented by a 3D Gaussian sphere. That is, the 3D objects in the target 3D scene are represented by a Gaussian sphere, and the parameters of the 3D Gaussian sphere are used as the learning objects of the target semantic prediction model. In this way, it is possible to avoid using a large amount of memory to store the complex scene being rendered, reduce memory consumption, and thus improve the computation speed of the model.

[0020] In one possible implementation of the first aspect, the process of determining the true language features of the aforementioned sample image may specifically include: an electronic device inputting the sample image into an image segmentation model to obtain an image segmentation result of the sample image; wherein the image segmentation result includes a set of pixels corresponding to each object in the sample image. Subsequently, the electronic device may input the set of pixels corresponding to multiple objects in the sample image into a visual language model to obtain the true language features of the sample image.

[0021] In this application, the electronic device uses image segmentation and visual language models to extract the real language features of sample images as supervision data for training the semantic prediction model. This means the electronic device can obtain the real language features of a sample image based solely on the sample image. This not only avoids relying on manually labeled semantic tags during training, saving significant time and manpower costs and improving the training efficiency of the semantic prediction model, but also allows the supervision data to be unrestricted by manually labeled semantic tags, enabling semantic prediction for any open vocabulary. This enhances the application flexibility and semantic prediction performance of the target semantic prediction model, providing a foundation for accurate subsequent detection of target objects.

[0022] In one possible implementation of the first aspect, the process of adjusting the parameters of the initialized three-dimensional Gaussian sphere by the electronic device may specifically include: the electronic device extracting features from the sample image using a feature extraction network to obtain two-dimensional image features of the sample image; wherein the two-dimensional image features include pixel features of each pixel in the sample image. Then, the electronic device can fuse the initialized three-dimensional Gaussian sphere and the two-dimensional image features of the sample image to obtain three-dimensional Gaussian sphere features. Subsequently, the electronic device can update the three-dimensional Gaussian sphere features using a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample image.

[0023] In this application, the electronic device can accurately determine the features of a three-dimensional Gaussian sphere by fusing and updating two-dimensional feature points with three-dimensional feature points, thereby providing a foundation for subsequent training of the target semantic prediction model.

[0024] In one possible implementation of the first aspect, the electronic device fuses the initialized three-dimensional Gaussian sphere with the two-dimensional image features of the sample image. Specifically, this process may include: the electronic device determining the pixel features corresponding to each Gaussian sphere from the two-dimensional image features based on the intrinsic and extrinsic parameters of the camera; wherein the camera is the one that captures the sample image. Then, for each Gaussian sphere in the three-dimensional Gaussian sphere, the electronic device can input the Gaussian sphere and its corresponding pixel features into a cross-attention network to obtain Gaussian sphere features. Finally, the electronic device can merge the Gaussian sphere features of multiple Gaussian spheres to obtain the three-dimensional Gaussian sphere features.

[0025] In this application, after obtaining the pixel features corresponding to each Gaussian sphere, the electronic device can utilize a cross-attention network to fuse the parameters of the Gaussian sphere with its corresponding pixel features to obtain sub-Gaussian sphere features. Subsequently, the electronic device can merge the sub-Gaussian sphere features corresponding to multiple Gaussian spheres to obtain three-dimensional Gaussian sphere features. This enables adaptive selection and fusion of multi-view image features, allowing the cloud service to fuse sub-image features with the three-dimensional Gaussian sphere from different perspectives, enhancing the richness of pixel-level representation and providing a foundation for subsequent training of semantic prediction models.

[0026] In one possible implementation of the first aspect, the process of updating the three-dimensional Gaussian sphere features by the electronic device may specifically include: if the sample database includes a sample image, or if the three-dimensional Gaussian sphere features are features obtained when training the semantic prediction model for the first time, the electronic device may input the three-dimensional Gaussian sphere features into a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample image.

[0027] In this application, an electronic device inputs the features of a three-dimensional Gaussian sphere into a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample image. This enables adaptive selection of Gaussian sphere features, thereby effectively acquiring the long-range dependencies between three-dimensional Gaussian spheres and providing a foundation for subsequent training of the semantic prediction model.

[0028] In one possible implementation of the first aspect, the process of updating the three-dimensional Gaussian sphere features by the electronic device may further include: when the sample database includes at least two sample images and the three-dimensional Gaussian sphere features are not features obtained when the semantic prediction model was trained for the first time, the electronic device may input the three-dimensional Gaussian sphere features and historical three-dimensional Gaussian sphere features into a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample images; wherein, the historical three-dimensional Gaussian sphere features are features obtained when the semantic prediction model was trained for the previous time.

[0029] In this application, the electronic device inputs the 3D Gaussian sphere features and historical 3D Gaussian sphere features into a self-attention network to obtain the target 3D Gaussian sphere features of the sample image. This enables adaptive selection and fusion of Gaussian sphere features, thereby effectively obtaining the long-range dependencies between 3D Gaussian spheres and providing a foundation for subsequent training of the semantic prediction model.

[0030] In one possible implementation of the first aspect, after the electronic device obtains the target three-dimensional Gaussian sphere features of the sample image, the method further includes: the electronic device inputting the target three-dimensional Gaussian sphere features into a Gaussian splash model to obtain the three-dimensional raster features of the sample image; wherein the three-dimensional raster features of the sample image are used to characterize the semantic information of the three-dimensional scene to which the sample image belongs.

[0031] In this application, after obtaining the aforementioned three-dimensional Gaussian sphere features of the target, the electronic device can use a Gaussian splash model to convert the target three-dimensional Gaussian sphere features into three-dimensional grid features. This facilitates subsequent determination of the three-dimensional position information of the object to be queried based on the three-dimensional grid features, providing traversal conditions for the electronic device to determine the object's position.

[0032] Secondly, this application provides a semantic prediction model training method for use in electronic devices or cloud services. The method involves acquiring a sample database, which includes at least one sample image displaying at least one object. Then, for each sample image in the database, the parameters of an initialized 3D Gaussian sphere are adjusted according to the sample image to obtain the target 3D Gaussian sphere features of the sample image. Next, the target 3D Gaussian sphere features are rendered using a Gaussian splash model to obtain the predicted language features of the sample image. Finally, a pre-constructed semantic prediction model is trained based on the predicted language features and the true language features of the sample image to obtain the target semantic prediction model; wherein the true language features of the sample image are obtained based on the sample image.

[0033] In one possible implementation of the second aspect, the process of determining the true language features of the aforementioned sample image may specifically include: inputting the sample image into an image segmentation model to obtain the image segmentation result of the sample image; wherein, the image segmentation result includes the set of pixels corresponding to each object in the sample image. Then, inputting the set of pixels corresponding to multiple objects in the sample image into a visual language model to obtain the true language features of the sample image.

[0034] In one possible implementation of the second aspect, the process of adjusting the parameters of the initialized 3D Gaussian sphere may specifically include: extracting features from the sample image using a feature extraction network to obtain two-dimensional image features of the sample image; wherein the two-dimensional image features include the pixel features of each pixel in the sample image. Then, the initialized 3D Gaussian sphere and the two-dimensional image features of the sample image are fused to obtain the 3D Gaussian sphere features. Finally, the 3D Gaussian sphere features are updated using a self-attention network to obtain the target 3D Gaussian sphere features of the sample image.

[0035] In one possible implementation of the second aspect, the process of fusing the initialized 3D Gaussian sphere and the 2D image features of the sample image can specifically include: determining the pixel features corresponding to each Gaussian sphere from the 2D image features based on the intrinsic and extrinsic parameters of the camera; wherein the camera is the camera that captures the sample image. Then, for each Gaussian sphere in the 3D Gaussian sphere, the Gaussian sphere and its corresponding pixel features are input into a cross-attention network to obtain Gaussian sphere features. Finally, the Gaussian sphere features of multiple Gaussian spheres are merged to obtain the 3D Gaussian sphere features.

[0036] In one possible implementation of the second aspect, the process of updating the three-dimensional Gaussian sphere features may specifically include: if the sample database includes a sample image, or if the three-dimensional Gaussian sphere features are features obtained when training the semantic prediction model for the first time, inputting the three-dimensional Gaussian sphere features into a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample image.

[0037] In one possible implementation of the second aspect, the process of updating the three-dimensional Gaussian sphere features may further include: when the sample database includes at least two sample images and the three-dimensional Gaussian sphere features are not features obtained when the semantic prediction model was trained for the first time, inputting the three-dimensional Gaussian sphere features and historical three-dimensional Gaussian sphere features into a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample images; wherein, the historical three-dimensional Gaussian sphere features are features obtained when the semantic prediction model was trained for the previous time.

[0038] In one possible implementation of the second aspect, after obtaining the target three-dimensional Gaussian sphere features of the sample image, the method further includes: inputting the target three-dimensional Gaussian sphere features into a Gaussian splash model to obtain the three-dimensional raster features of the sample image; wherein, the three-dimensional raster features of the sample image are used to characterize the semantic information of the three-dimensional scene to which the sample image belongs.

[0039] Thirdly, this application provides an electronic device including a camera, one or more processors, and one or more memories; the one or more processors are coupled to the camera and the one or more memories; the camera is used to acquire a target panoramic image, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the method described above.

[0040] Fourthly, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0041] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0042] In a sixth aspect, a chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, wherein when the code is executed, the processor is used to execute the method described above.

[0043] The beneficial effects that the semantic prediction model training method described in the second aspect, the electronic device described in the third aspect, the computer-readable storage medium described in the fourth aspect, the computer program product described in the fifth aspect, and the chip described in the sixth aspect can achieve can be referred to the beneficial effects in the first aspect and any of its possible design methods, and will not be repeated here. Attached Figure Description

[0044] Figure 1 This is a schematic diagram illustrating the result of object detection performed by an electronic device according to an embodiment of this application;

[0045] Figure 2 A schematic diagram showing the result of object detection performed by another electronic device provided in an embodiment of this application;

[0046] Figure 3 A hardware structure diagram of an electronic device provided in an embodiment of this application;

[0047] Figure 4 A software structure diagram of an electronic device provided in an embodiment of this application;

[0048] Figure 5 A flowchart of an object position detection method provided in an embodiment of this application;

[0049] Figure 6 A schematic diagram illustrating the generation of a target surround view image provided in an embodiment of this application;

[0050] Figure 7 A schematic diagram of a three-dimensional occupying grid provided for an embodiment of this application;

[0051] Figure 8 This is a schematic diagram illustrating a scenario where an indoor robot determines the position of an object, provided as an embodiment of this application.

[0052] Figure 9 This is a schematic diagram of an object position detection process provided in an embodiment of this application;

[0053] Figure 10 A flowchart illustrating a semantic prediction model training method provided in this application embodiment;

[0054] Figure 11 This is a schematic diagram illustrating a feature conversion process provided in an embodiment of this application;

[0055] Figure 12 This is a schematic diagram illustrating the process of training a semantic prediction model, as provided in an embodiment of this application.

[0056] Figure 13 This application provides a schematic diagram illustrating the process of training a semantic prediction model based on a target image sequence in an embodiment of the present application.

[0057] Figure 14 A flowchart illustrating another semantic prediction model training method provided in this application embodiment;

[0058] Figure 15 This is a schematic diagram illustrating the training process of another semantic prediction model provided in an embodiment of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0060] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0061] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] With the continuous development of computer vision technology, object detection methods based on computer vision have been widely applied in many scenarios, such as pedestrian recognition in autonomous vehicles and object querying in indoor robots. However, in applications such as autonomous driving, robotics, and augmented reality (AR) navigation, this object detection method can only detect objects in images captured by electronic devices, obtaining bounding boxes for the target objects in the image. These bounding boxes represent the positional information of the target object. In other words, this object detection method can only detect the position of the target object in the image and cannot obtain voxel-level object shape information, meaning it cannot effectively avoid obstacles based on the target object's bounding box. A voxel is the smallest unit in 3D spatial segmentation, equivalent to a 3D pixel.

[0063] For example, such as Figure 1 As shown, image 100 is an image captured by the camera device of an electronic device. This image 100 is used to represent a moving vehicle. Specifically, the electronic device can perform target detection on image 100 to obtain a detection box A for the vehicle in image 100. The detection box A is used to represent the vehicle's position information. For example, the electronic device can mark the vehicle in image 100 using detection box A, and the position information of the vehicle is represented by the coordinates corresponding to the four vertices of detection box A.

[0064] It's understandable that in real-world 3D scenes (or 3D space), target objects have 3D shapes, and most applications require parameters such as the object's length, width, height, and rotation angle. For example, in autonomous driving scenarios, it's crucial to provide the 3D size and rotation angle of the target object from the image, which is vital for subsequent path planning and control. Furthermore, while electronic devices using the aforementioned target detection methods can detect the location and category of a target object in an image, they cannot fill in occluded areas. This prevents the electronic device from making reasonable analyses based on the actual situation, affecting subsequent operations and ultimately impacting the user experience.

[0065] Therefore, to improve the accuracy of target object detection, electronic devices can utilize semantic prediction models to perform semantic prediction on acquired images, obtaining the semantic information of the image. This semantic information includes the category information corresponding to each pixel in the image. The semantic prediction model is trained on a labeled dataset, which includes multiple object images and the true semantic label of each object image. This true semantic label represents the true category information of the object in the image. Subsequently, the electronic device can determine the three-dimensional position information of the object belonging to the category information within the image based on the semantic information and the image itself.

[0066] It can be seen that the training process of this semantic prediction model relies on manually labeled semantic tags, wasting a significant amount of time and manpower. Furthermore, when using this semantic prediction model for object recognition, the electronic device can only identify objects in an image based on existing semantic tags in the tag database. In other words, the electronic device can match the object with existing semantic tags in the tag database; if an existing semantic tag in the tag database cannot match the object, the electronic device will not be able to obtain the object's semantic tag.

[0067] For example, taking an indoor robot as an example, indoor scenes contain a wide variety of objects, making it difficult to manually construct a labeled dataset for all objects. This means that indoor objects (such as slippers, blankets, etc.) may not be manually labeled, thus affecting the robot's object recognition performance and consequently impacting the user experience. For example, please see... Figure 2 Image 200 is a bedroom image captured by an indoor robot, which includes a bed, a table, and a speaker. Specifically, the indoor robot can use the semantic prediction model described above to identify objects and obtain the location information of the bed and the table in image 200. It can be understood that the category information "table," "bed," and "cabinet" represents the true category information of the objects in the labeled dataset during the training of the semantic prediction model; that is, the category information "table," "bed," and "cabinet" are manually pre-labeled.

[0068] In this embodiment, the semantic prediction model can represent pixels in image 200 that are categorized as "bed" using white, pixels categorized as "table" using diagonal lines, and pixels categorized as "cabinet" using intersecting lines. In other embodiments, the semantic prediction model can also represent pixels with different categories using different colors. For example, the semantic prediction model can represent pixels categorized as "chair" using black, and pixels categorized as "pillow" using pink, etc.

[0069] It should be noted that although the image 200 mentioned above also includes objects with the category information of "speaker", the label database does not include semantic labels with the category information of "speaker". Therefore, the indoor robot cannot obtain semantic information including the category information of "speaker", that is, it cannot obtain the position information of the speaker in the image 200. This results in low object detection accuracy of the indoor robot, which in turn affects the user experience.

[0070] Furthermore, the aforementioned semantic prediction model uses a dense grid to represent the 3D features of objects in a 3D scene, which in turn represents the object's position information in the 3D scene. However, since 3D features have a certain degree of sparsity in a real 3D scene, this will cause the 3D features to occupy too much storage space, resulting in a large memory consumption of electronic devices, which in turn affects the calculation speed of the semantic prediction model and ultimately affects the efficiency of determining the position information.

[0071] In one implementation, after acquiring the image, the electronic device extracts features from the image to obtain two-dimensional features. These two-dimensional features represent the positional information of the object in the image. Then, the electronic device can perform a viewpoint transformation on these two-dimensional features to obtain three-dimensional features. These three-dimensional features represent the positional information of the object in a three-dimensional scene.

[0072] In some embodiments, the electronic device can utilize a depth distribution estimation algorithm to perform viewpoint transformation on the aforementioned two-dimensional features. This depth distribution estimation algorithm uses probabilistic models and statistical methods to predict or estimate the distribution characteristics of depth data (such as the relative positions of objects indoors). While this can better predict the relative positions of objects indoors, making the predicted depth image closer to reality, it may result in sparse three-dimensional features and dependence on the accuracy of the depth distribution estimation algorithm. In other words, if the prediction performance of this depth distribution estimation algorithm is low, the accuracy of the three-dimensional features will be low, thus affecting the electronic device's ability to perform subsequent operations.

[0073] In other embodiments, the electronic device can perform perspective transformation on the aforementioned two-dimensional features through camera projection sampling. Specifically, the electronic device can project a three-dimensional object in three-dimensional space into a two-dimensional image using a camera device. Then, the electronic device can extract features from this two-dimensional image to obtain two-dimensional features. Subsequently, the electronic device can sample and interpolate these two-dimensional features to obtain three-dimensional features. While this method simplifies the acquisition of three-dimensional features, it may result in redundant or inaccurate features, leading to lower accuracy and potentially affecting subsequent operations performed by the electronic device.

[0074] In some embodiments, the electronic device can utilize a cross-attention mechanism to fuse the aforementioned two-dimensional features with initialized three-dimensional spatial features to obtain the aforementioned three-dimensional features. These three-dimensional spatial features are initialized by the electronic device based on three-dimensional objects in three-dimensional space. While this allows the cross-attention mechanism to have good adaptability, it typically requires a grid representation to characterize the initialized three-dimensional spatial features, resulting in excessive storage space consumption and high memory usage for the electronic device. This, in turn, affects the computational speed of the semantic prediction model and ultimately impacts the efficiency of location information determination.

[0075] Therefore, to improve the detection accuracy of the target object's location, this application provides an object location detection method. In this method, in response to a query command, an electronic device acquires a target surround view image. The query command includes object information of the object to be queried, and the target surround view image represents the three-dimensional scene of the environment in which the electronic device is located. Then, the electronic device can use a target semantic prediction model to perform semantic prediction on the target surround view image to obtain the semantic information of the three-dimensional scene. This semantic information includes object information corresponding to each voxel in the three-dimensional scene. The target semantic prediction model is trained based on a sample database, which includes at least one sample image. Then, the electronic device can determine the location information of the object to be queried in the three-dimensional scene based on the semantic information of the three-dimensional scene and the object information of the object to be queried.

[0076] In this embodiment, the electronic device can obtain object information corresponding to each voxel in the 3D scene to which the target surround view image belongs by performing semantic prediction. This allows the electronic device to determine the position of the target object in its environment. This enables accurate detection of the object's location, facilitating subsequent operations by the electronic device and improving the user experience. Furthermore, the target semantic prediction model can be trained solely on sample images, without relying on manually labeled semantic tags. This not only saves significant time and manpower costs, improving the training efficiency of the semantic prediction model, but also allows the model to perform semantic prediction on any open vocabulary, freeing it from the limitations of manually labeled tags. This enhances the application flexibility and semantic prediction performance of the target semantic prediction model, facilitating accurate object location detection.

[0077] For example, the aforementioned electronic device can be a robot, robotic arm, intelligent vehicle, AR device, or other device capable of collecting a surround-view image of the environment surrounding the electronic device. This application embodiment does not impose any special limitations on the specific form of the electronic device. The robot can be an indoor robot (such as a cleaning robot).

[0078] In some embodiments, taking an indoor robot as an example, the query command can be a command output by the user's voice or a command input by the user into the indoor robot or the target electronic device. The target electronic device is an electronic device that has a communication connection with the indoor robot; for example, it can be a mobile phone, tablet computer, etc., and there is no specific limitation.

[0079] In one example, after determining the location information of the object to be queried in a 3D scene (or the 3D location information of the object to be queried), the indoor robot can output a prompt message (or location prompt message) about the aforementioned 3D location information, so that the user can know the location of the object to be queried in a timely manner, reducing the query time, improving the query efficiency, and thus enhancing the user experience. This prompt message can be output through voice, text, or other means. For example, the indoor robot can output the voice prompt message "The cup is on the table in the bedroom." Alternatively, the indoor robot's display screen can show the text prompt message "The cup is on the table in the bedroom."

[0080] In another example, the query command can also include action information. Specifically, after determining the three-dimensional position of the object to be queried, the indoor robot can directly execute the corresponding action command. This improves the robot's action execution efficiency and thus enhances the user's interactive experience. For example, if the query command is "clean the carpet," the indoor robot can directly clean the carpet after determining its location.

[0081] In other embodiments, taking an electronic device as an example of a smart vehicle, the aforementioned query command can be a command output by the user's voice or a command pre-set by the smart vehicle. For example, the object to be queried in the query command can be a pedestrian, a vehicle, etc. Specifically, after determining the three-dimensional position information of the object to be queried, the smart vehicle can perform driving decisions such as route planning and obstacle avoidance based on the three-dimensional position information. For example, if the object to be queried is a pedestrian, and the pedestrian is directly in front of the smart vehicle, the smart vehicle can stop to avoid the pedestrian, or the smart vehicle can also drive around the pedestrian to continue driving.

[0082] Figure 3 This is a schematic diagram of the hardware structure of the electronic device 300 provided in the embodiments of this application, as shown below. Figure 3 As shown, the electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, a sensor 380, a button 390, a motor 391, an indicator 392, a camera 393, and a display screen 394, etc.

[0083] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0084] Processor 310 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0085] In some embodiments, the electronic device 300 may use the processor 310 to perform the object position detection method provided in this application.

[0086] The wireless communication function of electronic device 300 can be realized through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor and baseband processor, etc.

[0087] Electronic device 300 implements display functions through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0088] The display screen 394 is used to display images, videos, etc. In some embodiments, the electronic device 300 can display prompts about the location of the object to be queried through the display screen 394, so that the user can know the location of the object to be queried in a timely manner, reduce the query time, improve the query efficiency, and thus improve the user experience.

[0089] Electronic device 300 can achieve shooting function through ISP, camera 393, video codec, GPU, display 394 and application processor.

[0090] Electronic device 300 can implement audio functions such as music playback and recording through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor.

[0091] The speaker 370A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. In some embodiments, the electronic device 300 can use the speaker 370A to output voice prompts about the location of the object to be queried.

[0092] Microphone 370C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. In some embodiments, electronic device 300 can acquire user-input query commands via microphone 370C. These query commands include the object to be queried, which is the object the user needs to find. Electronic device 300 can then perform feature extraction on the query command to obtain the linguistic features of the object to be queried. These linguistic features are used to characterize the object information of the object to be queried.

[0093] Button 390 may include a power button, volume buttons, etc. Button 390 may be a mechanical button or a touch button. Electronic device 300 can receive button input and generate key signal inputs related to user settings and function control of the electronic device. Motor 391 can generate vibration alerts. Motor 391 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 392 may be an indicator light, used to indicate charging status, battery level changes, or to indicate messages, missed calls, notifications, etc.

[0094] For example, the software system of the aforementioned electronic device 300 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to illustrate the software structure of the electronic device 300.

[0095] Figure 4 This is a software structure block diagram of the electronic device 300 according to an embodiment of this application.

[0096] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer (application layer), the application framework layer (framework layer), the Android runtime and system libraries, and the kernel layer (or driver layer). The application layer can include a series of application packages.

[0097] The application layer can include a series of application packages. This application layer can include multiple application packages. For example... Figure 4 As shown, the application package can include a camera, an object recognition module, an action execution module, etc.

[0098] The object recognition module is used to receive query commands output by the user and to control the camera (or imaging device) to acquire a surround view image of the target. In some embodiments, upon receiving a query command output by the user, the object recognition module may send an image acquisition command to the camera so that the camera can acquire a surround view image of the target. In other embodiments, upon receiving a query command output by the user, the object recognition module may send the query command output by the user to an instruction processing module so that the instruction processing module can process the query command.

[0099] In one implementation, upon receiving a 3D mask of the object to be queried, the object recognition module can obtain the 3D position information of the object based on the mask. This 3D position information represents the position of the object within the environment of the electronic device. Subsequently, the object recognition module can output location hints, which also represent the 3D position information of the object.

[0100] The aforementioned action execution module is used to perform a target operation based on the three-dimensional position information of the object to be queried. This target operation can be pre-set by the electronic device or can be action information included in the query command. For example, the target operation can be a path planning operation, obstacle avoidance operation, cleaning operation, etc., and is not specifically limited. In some embodiments, upon receiving the three-dimensional position information sent by the object recognition module, the action execution module can execute the target operation corresponding to the object to be queried. For example, if the query command is "clean the carpet," after receiving the three-dimensional position information of the carpet, the action execution module can clean the carpet when the electronic device reaches its location, thereby completing the carpet cleaning task and improving the user's interactive experience.

[0101] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 4 As shown, the application framework layer may include a window manager service (WMS), a content provider, a notification manager, a resource manager, and a view system.

[0102] The window manager manages the windowed applications. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc. The notification manager allows applications to display notifications in the status bar, conveying informative messages; these notifications can disappear automatically after a short pause without user interaction. The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. The view system includes visual controls, such as controls for displaying text and images. The view system is used to build applications. The display interface can consist of one or more views.

[0103] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0104] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0105] The system library can include multiple functional modules, such as an instruction processing module, a semantic prediction module, a feature matching module, and media libraries. The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0106] The instruction processing module processes user-input query instructions. In some embodiments, upon receiving a query instruction, the instruction processing module can identify the object information of the object to be queried in the query instruction to obtain query text. Then, the instruction processing module can input the query text into a visual language model to obtain the linguistic features of the object to be queried. These linguistic features characterize the object information of the object to be queried. This object information includes category information and / or attribute information. The category information characterizes the category to which the object to be queried belongs, that is, it is used to classify objects with common characteristics. For example, the category information can be table, chair, etc. The attribute information characterizes the specific characteristics of the object to be queried. For example, the attribute information can be color, material, weight, etc.

[0107] The semantic prediction module is used to perform semantic prediction on the aforementioned target surround view image. In some embodiments, upon receiving the target surround view image, the semantic prediction module can perform semantic prediction on the target surround view image to obtain the semantic information of the 3D scene to which the target surround view image belongs. This semantic information includes object information corresponding to each voxel in the 3D scene. In this embodiment, the semantic information of the 3D scene can be represented by 3D raster features (or 3D voxel features), that is, the 3D raster features include raster features (or voxel features) of multiple rasters, and each raster feature corresponds to the object information of a voxel in the 3D scene. It can be understood that one raster is equivalent to one unit, that is, equivalent to one voxel.

[0108] The feature matching module matches 3D raster features with the linguistic features of the query object to determine the 3D mask of the query object. In some embodiments, for each raster feature in the 3D raster features, the feature matching module can calculate the similarity between the raster feature and the linguistic features. Then, the feature matching module can assemble the raster features with similarity greater than a preset similarity into a 3D mask of the query object. This 3D mask of the query object represents the positional information of the query object in the 3D scene, that is, it represents the 3D positional information of the query object.

[0109] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, and audio drivers.

[0110] In some embodiments, such as Figure 4 As shown, in response to a user's query command, the electronic device (or alternatively described as an object recognition module) can execute step a, triggering the command processing module in the system library to extract features from the query command and obtain the linguistic features of the object to be queried. Then, the command processing module can execute step b, sending the linguistic features of the object to be queried to the feature matching module in the system library for subsequent feature matching.

[0111] Furthermore, the electronic device can execute step c, triggering the camera to capture a target surround view image. Then, the camera can execute step d, sending the captured target surround view image to a system library (such as a semantic prediction module within the system library), enabling the semantic prediction module to perform semantic prediction on the target surround view image and obtain 3D raster features. Next, the semantic prediction module can execute step e, triggering the feature matching module in the system library to match the raster features in the 3D raster features with the linguistic features of the object to be queried, to obtain a 3D mask of the object to be queried. Finally, the semantic prediction module can execute step f, sending the 3D mask of the object to be queried to an object recognition module, enabling the object recognition module to determine the 3D position information of the object to be queried.

[0112] Optionally, the object recognition module can perform step g, sending the three-dimensional position information of the object to be queried to the action execution module, so that the action execution module can perform the target operation, thereby improving the action execution efficiency of the electronic device and thus enhancing the user's interactive experience.

[0113] Understandable, Figure 4 The layers in the illustrated structure and the components contained in each layer do not constitute a specific limitation on the electronic device 300, i.e., the robot or intelligent vehicle. In other embodiments of this application, the structure may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0114] The object position detection method of this application can be applied to any scenario where object position is detected. For example, an electronic device can capture a surround view image of a target using its camera, and determine the three-dimensional position information of the target object (or the object to be queried) based on the surround view image, that is, determine the position information of the target object in a three-dimensional scene. In the following embodiments, the method of this application is described using an indoor robot as an example, where the electronic device is an object position detection scenario.

[0115] Specifically, this application provides an object position detection method. After an indoor robot acquires a target surround view image through a camera, it can use a target semantic prediction model to perform semantic prediction on the target surround view image to obtain three-dimensional grid features. Then, the indoor robot can match these three-dimensional grid features with the linguistic features of the target object to obtain the target object's three-dimensional mask features, thereby determining the target object's three-dimensional position information, that is, determining the target object's position information in the space where the indoor robot is located, thus improving the target object recognition accuracy. For example, as shown... Figure 5 As shown, the object position detection method may include S501 to S504:

[0116] S501, in response to the user's query command, the indoor robot acquires a surround view image of the target. The query command includes object information about the target object.

[0117] Specifically, the indoor robot can receive query commands from the user in real time. These commands trigger the robot to search for objects based on the object information provided and output the 3D position information of the target object. In other words, the query command can include object information about the target object, which is the object the user wants to query.

[0118] The object information mentioned above may include category information and / or attribute information. Category information is used to characterize the category to which the target object belongs, that is, to categorize objects with common characteristics. In other words, the category information of the target object is predetermined and unchanging. For example, the category information for tables and chairs can be furniture, and the category information for shirts and short-sleeved shirts can be clothing. That is, if a query instruction includes the category information of the target object, the query instruction can be "find clothes," "find cups," etc. Attribute information is used to characterize the specific features of the target object. That is, the attribute information can be determined based on the characteristics of the target object. For example, the attribute information can be the object's color, material, etc. For example, the color of a water cup can be pink, red, etc., and the material of a water cup can be stainless steel, glass, etc. That is, if a query instruction includes the attribute information of the target object, the query instruction can be "find red objects," "find glass objects," etc.

[0119] It is understandable that, since the aforementioned object information can include not only attribute information and / or category information representing fixed words, but also attribute information and / or category information representing open words, the indoor robot can query the 3D position information of the corresponding object for any open word, without being limited to any fixed word in the fixed word list. The fixed word list is a fixed set of labels, that is, the semantic labels already existing in the label database when training the semantic prediction model. The open word is an extensible set of labels, which allows the computer vision system to update and learn new labels when encountering new objects or scenes. In other words, the semantic prediction model trained with open words can better adapt to the diversity of the real world. For example, if the above query command includes the attribute information that the object is red, the indoor robot can determine the position information of the target object with the red color based on the above target surround view image. In this way, the target object can be queried more comprehensively, reducing the occurrence of situations where the target object cannot be found due to the voice command being an open word, improving the detection accuracy of the target object, and thus improving the user experience.

[0120] In one implementation, the query instruction can be a user-output language command. This language command may include wake-up information (or a wake-up word) and object information of the target object. The wake-up information is used to indicate the device the user wants to control, that is, the device that needs to respond to the query instruction and perform the corresponding operation. In this embodiment, the wake-up information is the wake-up word corresponding to an indoor robot (such as the robot Youyou). In other embodiments, the wake-up information may also be the wake-up word corresponding to other electronic devices with voice wake-up functions (such as tablets, smart speakers, etc.), etc., without specific limitations. It is understood that the same wake-up word corresponds to only one electronic device; that is, different electronic devices have different wake-up words, so that the voice assistant can determine the device the user wants to wake up based on the wake-up word. Furthermore, the wake-up word corresponding to the electronic device can be preset by the user according to their own habits, or it can be a default word for each electronic device, as long as it can distinguish each electronic device, without specific limitations.

[0121] In another implementation, the query command can be a text command entered by the user on the indoor robot. Specifically, in response to the user's command input operation on the indoor robot, the indoor robot can obtain the corresponding query command based on the command input operation. This command input operation can be a touch operation by the user on the input buttons displayed on the indoor robot's touchscreen, or a click operation by the user on the mechanical input buttons on the indoor robot, etc., and is not specifically limited.

[0122] In another implementation, the query command can be a text command entered by the user on the target electronic device. This target electronic device is one that establishes a communication connection with the indoor robot; that is, the target electronic device and the indoor robot share an intelligent voice sharing system. This intelligent voice sharing system is used to share query commands received by all electronic devices. In other words, as long as one electronic device in the intelligent voice sharing system can receive a query command from the user, that electronic device can control the indoor robot to perform the corresponding action based on the query command.

[0123] In some embodiments, electronic devices can establish a smart voice sharing system through an application (APP). In one example, if the electronic device is a device with a display screen (such as a mobile phone, computer, etc.), the user can download a smart voice sharing APP from the electronic device (e.g., an app store within the electronic device). The user can then log in to the same account within the smart voice sharing APP to establish the smart voice sharing system.

[0124] Furthermore, it should be understood that electronic devices in an intelligent voice sharing system can communicate with each other through a preset wireless communication method. This preset wireless communication method may include one or more of Bluetooth, Wi-Fi, NFC, and mobile networks (such as 2G, 3G, 4G, 5G, etc.).

[0125] Specifically, upon receiving the aforementioned query command, the indoor robot can capture a target surround view image via its camera. This target surround view image is a 3D scene image including the environment in which the indoor robot is located. In other words, this target surround view image is used to display the environment around the indoor robot. The indoor robot can observe its surroundings through this target surround view image to quickly detect the location information of target objects, thereby improving the user experience.

[0126] In some embodiments, the indoor robot may be equipped with cameras in four directions (i.e., front, rear, left, and right). After obtaining local scene images captured by these four cameras, the indoor robot can process and synthesize these four local scene images to obtain the aforementioned target panoramic image. The four cameras are RGB cameras including panoramic lenses. These panoramic lenses can capture a field of view greater than or equal to 180 degrees. For example, as shown... Figure 6 As shown, after the indoor robot acquires four partial scene images through cameras in four directions, it can process and synthesize these four partial scene images to obtain a target surround view image. It can be seen that this target surround view image displays the environment surrounding the indoor robot.

[0127] In one implementation, the indoor robot can acquire a target surround view image in real time. Then, upon receiving the aforementioned query command, the indoor robot can determine the target object's position information in the 3D scene based on the target surround view image. That is, in this embodiment, the acquisition node of the target surround view image is not displayed; the target surround view image can be acquired before or after receiving the query command, without any specific limitation.

[0128] S502, the indoor robot uses a target semantic prediction model to perform semantic prediction on the target surround view image and obtain three-dimensional grid features.

[0129] In some embodiments, after obtaining the aforementioned target surround view image, the indoor robot can perform semantic prediction on the target surround view image to obtain the three-dimensional semantic information of the three-dimensional scene to which the target surround view image belongs. This three-dimensional semantic information is used to characterize the object information corresponding to each voxel in the three-dimensional scene. This three-dimensional semantic information can be represented using three-dimensional raster features.

[0130] Specifically, after obtaining the aforementioned target surround view image, the indoor robot can input the target surround view image into the target semantic prediction model to obtain the 3D grid features of the 3D scene. These 3D grid features are used to characterize the object information corresponding to each voxel in the 3D scene. In this embodiment, the indoor robot can divide the environment in which it is located into a preset number of grids; that is, the 3D grid features include grid features of multiple grids. It can be understood that one grid corresponds to one voxel in the 3D scene. If a grid is occupied, it indicates that there are obstacles in the environment corresponding to that grid; if a grid is idle, it indicates that there are no obstacles in the environment corresponding to that grid.

[0131] The aforementioned target semantic prediction model is used to predict the object information to which an object belongs in a 3D scene, thereby obtaining the object information corresponding to each voxel in the 3D scene. The training process of this target semantic prediction model will be described in detail below, and will not be elaborated upon here.

[0132] S503, the indoor robot extracts features from the query command to obtain the language features of the target object.

[0133] Specifically, after receiving the query command, the indoor robot can recognize the query command and obtain the query text. Then, the indoor robot can extract features from the query text to obtain the linguistic features of the target object. These linguistic features are used to characterize the object information of the target object.

[0134] In one implementation, the indoor robot can input the query text into a trained visual language model to obtain the language features of the target object.

[0135] It should be noted that the semantic prediction process of the target surround view image and the feature extraction process of the query command can be executed simultaneously or sequentially, without any specific limitation. For example, the indoor robot can execute steps S501-S502 and step S503 simultaneously. Alternatively, after receiving the query command, the indoor robot can first execute step S503, then execute "collect target surround view image" in S501, and then step S502.

[0136] S504, the indoor robot matches the language features of the target object with the three-dimensional grid features to obtain the three-dimensional position information of the target object.

[0137] Specifically, after obtaining the linguistic features and 3D grid features of the target object, the indoor robot can sequentially match the grid features in the 3D grid features with the linguistic features of the target object to obtain the 3D position information of the target object. This 3D position information refers to the target object's position in the 3D scene, that is, its position within the indoor robot's environment. This enables accurate detection of the object's location, facilitating subsequent operations by electronic devices and thus improving the user experience.

[0138] It is understood that the above feature matching results can be represented by a 3D mask. This 3D mask is used to characterize the positional information of the target object within the environment in which the indoor robot is located. It is understood that this 3D mask is a subset of the grid features in a 3D grid. For example, such as... Figure 7 As shown, the multiple grids corresponding to the black area in the 3D occupancy grid 700 are 3D masks, that is, the black area is the voxel range occupied by the target object.

[0139] In some embodiments, for each of the aforementioned three-dimensional grid features, the indoor robot calculates the similarity between that grid feature and the linguistic features of the target object. Then, the indoor robot can merge grid features with a similarity greater than a preset similarity (or a first preset similarity) into a three-dimensional mask to obtain the three-dimensional position information of the target object. In this embodiment, the indoor robot determines the position information of the target object in its environment by calculating the similarity between features. This means the indoor robot can detect the position of the target object in real-time and accurately during its movement, improving the accuracy of real-time object position detection and enabling users to quickly find the target object, thus enhancing the user experience.

[0140] For example, taking the target object as a red object, if the similarity between any grid feature and the language feature of the red object is greater than a preset similarity, it means that the grid corresponding to the grid feature can represent the object information of the red object. Therefore, the indoor robot can merge the grid features with a similarity greater than the preset similarity into a three-dimensional mask to obtain the position information of the red object in the three-dimensional scene.

[0141] The above describes how an indoor robot determines the 3D position information of a target object from a target surround view image based on a received query command. The following section will further describe one possible implementation process for an indoor robot to determine the 3D position information of a target object based on query commands. For example... Figure 8As shown, the process can specifically include: upon receiving a user's query instruction (e.g., "Hello Youyou, find the red object"), the mobile phone can send the query instruction to the indoor robot. Then, upon receiving the query instruction, the indoor robot acquires a target surround view image. Subsequently, the indoor robot can determine the three-dimensional position information of the target object based on the target surround view image and the object information (i.e., the red object) included in the query instruction.

[0142] It is understood that the aforementioned mobile phone can serve as a target electronic device to receive voice commands output by the user. Specifically, this target electronic device is one that has a communication connection with the indoor robot.

[0143] In some embodiments, upon receiving a user's query command, the indoor robot can directly acquire a target surround view image. Then, the indoor robot can determine the three-dimensional position information of the target object based on the target surround view image and the object information included in the query command. In other words, the indoor robot can directly receive the user's query command without needing to receive and send it through the target electronic device. This improves the efficiency of the indoor robot's object query, thereby enhancing the user experience.

[0144] The above described the process of an indoor robot querying objects based on query commands. The following section will introduce a possible implementation process for an indoor robot to determine the 3D mask of a target object to obtain the target object's position information within the robot's environment. For example... Figure 9 As shown, the process can specifically include: when the indoor robot receives the above query instruction, it can use the target visual language model to extract features from the query instruction and obtain the language features of the target object.

[0145] Furthermore, the indoor robot can acquire a surround-view image of the target. Then, the indoor robot can input this surround-view image into a target semantic prediction model to obtain 3D raster features. Next, the indoor robot can calculate the similarity between each raster feature and the linguistic features of the target object, obtaining multiple similarity scores for each raster feature. Finally, the indoor robot can select raster features with similarity scores greater than a preset similarity score from these multiple raster features and merge these raster features into a 3D mask of the target object.

[0146] In one implementation, the aforementioned target semantic prediction model can be trained and deployed by an indoor robot to detect the position information of the target object. In other words, the indoor robot is the primary entity executing the training process of this target semantic prediction model.

[0147] In another implementation, to reduce the cost and improve the efficiency of model training, the aforementioned target semantic prediction model can be trained using cloud services. That is, after obtaining the target semantic prediction model, the cloud service can send it to the indoor robot, effectively deploying the model within the robot. This enables timely detection of object positions, allowing users to quickly locate target objects and improving the user experience.

[0148] In one implementation, the execution entity is a cloud service, and the training process of the aforementioned target semantic prediction model is described in detail. Specifically, such as... Figure 10 As shown, the training process of this target semantic prediction model may include S801 to S806:

[0149] S801, cloud service obtains training dataset.

[0150] The training dataset includes at least one sample image, which may include at least one object. For example, the sample image may be a panoramic view of the environment in which the indoor robot is located, that is, a local scene image taken from any angle, as long as the sample image includes an object, there is no specific limitation.

[0151] It is understandable that, since the aforementioned training dataset includes only at least one sample image, the cloud service can train the pre-built semantic prediction model using only this single sample image to obtain the target semantic prediction model, without relying on manually labeled semantic tags. This not only saves significant time and manpower costs, improving the training efficiency of the semantic prediction model, but also allows the target semantic prediction model to be unrestricted by manually labeled semantic tags, enabling semantic prediction for any open vocabulary. This enhances the application flexibility and semantic prediction performance of the target semantic prediction model, facilitating accurate object location detection.

[0152] S802: For each sample image in the training dataset, the cloud service adjusts the parameters of the initialized 3D Gaussian sphere according to the sample image to obtain the target 3D Gaussian sphere features of the sample image.

[0153] Specifically, after obtaining the aforementioned training dataset, the cloud service can initialize the parameters of the 3D Gaussian sphere for each sample image in the training dataset to obtain an initialized 3D Gaussian sphere. The parameters of this 3D Gaussian sphere are randomly assigned by the cloud service. In other words, the parameters of the 3D Gaussian sphere are randomly set. It can be understood that this 3D Gaussian sphere comprises multiple Gaussian spheres, and each Gaussian sphere can correspond to one or more voxels in a 3D occupying grid.

[0154] In this embodiment, the three-dimensional scene model adapted to the sample three-dimensional scene (or target three-dimensional scene) is represented by the three-dimensional Gaussian sphere. That is, the three-dimensional objects in the sample three-dimensional scene are represented by the Gaussian sphere, and the parameters of the three-dimensional Gaussian sphere are used as the learning objects of the target semantic prediction model. In this way, it is possible to avoid using a large amount of memory to store the complex scene being rendered, reduce memory consumption, and thus improve the calculation speed of the model.

[0155] Furthermore, the cloud service can utilize a feature extraction network to extract features from the aforementioned sample image, obtaining a two-dimensional image feature of the sample image. This two-dimensional image feature includes the pixel features of each pixel in the sample image; that is, the two-dimensional image feature includes multiple pixel features. For example, the feature extraction network is a convolutional neural network.

[0156] In some embodiments, after obtaining the aforementioned two-dimensional image features and the initialized three-dimensional Gaussian sphere, the cloud service can determine the pixel features corresponding to each Gaussian sphere from the two-dimensional image features based on the camera's intrinsic and extrinsic parameters. The camera is the one that captured the aforementioned sample image, and its intrinsic and extrinsic parameters are used to obtain three-dimensional scene information from the sample image to achieve three-dimensional reconstruction.

[0157] It is understandable that, since the sample images are two-dimensional images, while the scenes displayed in the sample images are three-dimensional scenes (meaning the objects included in the sample images are three-dimensional objects in the real-world scene), the cloud service needs to use a 3D-2D projection matrix to render the three-dimensional scene onto the two-dimensional image (i.e., the sample image) to obtain the pixel features corresponding to each Gaussian sphere. In other words, this 3D-2D projection matrix is ​​used to project three-dimensional points in the camera coordinate system onto two-dimensional pixel coordinates, thereby obtaining the pixel features corresponding to each Gaussian sphere.

[0158] However, when rendering a two-dimensional image of the aforementioned three-dimensional scene using a 3D-2D projection matrix, image rendering is required based on the intrinsic and extrinsic parameters of the aforementioned camera. This ensures that the imaging points in the rendered sample image correspond completely to the spatial points in the sample three-dimensional scene. This facilitates the cloud service in reconstructing a three-dimensional scene model that is compatible with the sample three-dimensional scene based on the sample image, thus providing a foundation for subsequent training of the semantic prediction model.

[0159] The intrinsic parameters of the camera are used to characterize its internal features. For example, the intrinsic parameters of the camera are the relative positional relationships between pixels in the sample image and three-dimensional points in the camera coordinate system. In other words, the cloud service can use the intrinsic parameters of the camera to map three-dimensional points in the camera coordinate system to two-dimensional pixel coordinates on the image plane, so that the pixels in the sample image correspond to three-dimensional points in the camera coordinate system. The extrinsic parameters of the camera are used to characterize the camera's position and orientation. For example, the extrinsic parameters of the camera are the relative positional relationships between spatial points in the world coordinate system and three-dimensional points in the camera coordinate system. In other words, the cloud service can use the extrinsic parameters of the camera to convert spatial points in the world coordinate system into three-dimensional points in the camera coordinate system.

[0160] Specifically, after obtaining the pixel features corresponding to each Gaussian sphere, the cloud service can use a cross-attention network to fuse the parameters of the Gaussian sphere with its corresponding pixel features to obtain sub-Gaussian sphere features. Then, the cloud service can merge the sub-Gaussian sphere features corresponding to multiple Gaussian spheres to obtain three-dimensional Gaussian sphere features. The cross-attention network is used to obtain the correlation between two-dimensional image features and the three-dimensional Gaussian sphere.

[0161] In some embodiments, the cloud service can input the parameters of the Gaussian sphere as a query (Q) vector, and the corresponding pixel features of the Gaussian sphere as key (K) and value (V) vectors into the cross-attention network to obtain the 3D Gaussian sphere features. This enables adaptive selection and fusion of multi-view image features, allowing the cloud service to fuse sub-image features with the 3D Gaussian sphere from different perspectives, enhancing the richness of pixel-level representation and providing a foundation for subsequent training of semantic prediction models.

[0162] Specifically, after obtaining the aforementioned 3D Gaussian sphere features, if the training dataset includes at least two sample images, and the 3D Gaussian sphere features are not features obtained during the first training of the semantic prediction model, the cloud service can utilize a self-attention network to fuse these 3D Gaussian sphere features with historical 3D Gaussian sphere features to obtain the target 3D Gaussian sphere features. The historical 3D Gaussian sphere features are those fused before these 3D Gaussian sphere features, i.e., those fused during the previous training of the semantic prediction model. The self-attention network is used to handle the interactions between different Gaussian spheres, enabling any Gaussian sphere to perceive the sub-Gaussian sphere features of other Gaussian spheres, thereby effectively acquiring the long-range dependencies between Gaussian spheres.

[0163] In some embodiments, the cloud service can input the aforementioned Gaussian sphere features and historical Gaussian sphere features into a self-attention network to obtain fused Gaussian sphere features, i.e., target Gaussian sphere features, thereby updating the Gaussian sphere features. This enables adaptive selection and fusion of Gaussian sphere features, effectively obtaining the long-range dependencies between 3D Gaussian spheres and providing a foundation for subsequent training of the semantic prediction model.

[0164] It is understandable that if the above training dataset only includes one sample image, or if the above three-dimensional Gaussian sphere feature is the feature obtained when training the semantic prediction model for the first time, the cloud service can input only the three-dimensional Gaussian sphere feature into the self-attention network to obtain the target three-dimensional Gaussian sphere feature.

[0165] It should be noted that after obtaining the aforementioned 3D Gaussian sphere features of the target, the cloud service can utilize a Gaussian splash model to convert these features into 3D grid features. These 3D grid features are used to characterize the semantic information of the 3D scene. This facilitates subsequent determination of the target object's 3D position information based on these grid features, providing traversal conditions for indoor robots to determine object positions. For example, as shown... Figure 11 As shown, the cloud service inputs the target 3D Gaussian sphere feature 10A into the Gaussian splash model to obtain the 3D raster feature 10B.

[0166] S803, the cloud service uses the Gaussian splash model to render the three-dimensional Gaussian sphere features of the target and obtain the predicted language features of the sample image.

[0167] Specifically, after obtaining the aforementioned 3D Gaussian sphere features of the target, the cloud service can input these features into the Gaussian splash model to obtain the predicted language features of the sample image. These predicted language features are used to characterize the predicted object information corresponding to each pixel in the sample image.

[0168] S804, the cloud service uses an image segmentation model to segment the sample image and obtain the image segmentation result. This image segmentation result includes the set of pixels corresponding to each object in the sample image.

[0169] In some embodiments, after obtaining the training dataset, the cloud service can input the sample images from the training dataset into the image segmentation model to obtain the image segmentation result. This image segmentation result includes the set of pixels corresponding to each object in the sample image.

[0170] It can be understood that the above image segmentation model is a pre-trained image segmentation model.

[0171] S805, the cloud service uses a visual language model to extract features from the set of pixels corresponding to multiple objects in a sample image, thus obtaining the language features of the sample image.

[0172] Specifically, after obtaining the image segmentation results described above, the cloud service can input the set of pixels corresponding to multiple objects in the sample image into a visual language model to obtain the true language features of the sample image. These true language features are used to characterize the real object information corresponding to each pixel in the sample image.

[0173] It is understood that the above visual language model is a pre-trained visual language model.

[0174] In this embodiment, the cloud service uses image segmentation and visual language models to extract the real language features of sample images, which are then used as supervision data for training the semantic prediction model. This means the cloud service can obtain the real language features of a sample image based solely on the sample image. This not only avoids relying on manually labeled semantic tags during training, saving significant time and manpower costs and improving the training efficiency of the semantic prediction model, but also allows the supervision data to be unrestricted by manually labeled semantic tags, enabling semantic prediction for any open vocabulary. This enhances the application flexibility and semantic prediction performance of the target semantic prediction model, providing a foundation for accurate subsequent detection of target objects.

[0175] It should be noted that the processes for determining predicted language features and determining actual language features described above can be performed simultaneously or sequentially, without any specific limitation. For example, the cloud service can execute steps S802-S803 simultaneously, as well as steps S804-S805. Alternatively, the cloud service can execute steps S804-S805 first, and then execute steps S802-S803.

[0176] S806, the cloud service trains the pre-built semantic prediction model based on the predicted language features and the real language features to obtain the target semantic prediction model.

[0177] Specifically, after obtaining the real language features and predicted language features of the sample images, the cloud service can iteratively train the pre-built semantic prediction model based on the real language features and predicted language features to obtain the trained semantic prediction model, which is the target semantic prediction model.

[0178] In some embodiments, the cloud service can determine a loss value based on the aforementioned real language features and predicted language features. This loss value represents the error between the real language features and the predicted language features. Subsequently, the cloud service can adjust the parameters of the pre-built semantic prediction model based on this loss value until the loss value is less than a preset loss value, and the number of iterations to obtain the target semantic prediction model reaches a preset number of training iterations.

[0179] In one implementation, the cloud service can use gradient descent to adjust the parameters of the pre-built semantic prediction model. Gradient descent is an iterative optimization algorithm used to find the minimum value of the loss function. In other words, the cloud service can use gradient descent to minimize the loss function of the semantic prediction model, thereby adjusting the model parameters to gradually reduce the loss value.

[0180] The training process of the target semantic prediction model has been described above. The following section will further detail one possible implementation process for training a target semantic prediction model using cloud services. For example... Figure 12 As shown, the training process can specifically include: the cloud service acquiring sample images from the sample dataset. Then, the cloud service can initialize the parameters of the 3D Gaussian sphere to obtain an initialized 3D Gaussian sphere.

[0181] Furthermore, the cloud service can input the aforementioned sample image into a feature extraction network to obtain the two-dimensional image features of the sample image. Then, the cloud service can input the initialized three-dimensional Gaussian sphere and the two-dimensional image features of the sample image into a cross-attention network to obtain the three-dimensional Gaussian sphere features. Next, the cloud service can input this three-dimensional Gaussian sphere feature and historical three-dimensional Gaussian sphere features into a self-attention network to obtain the target three-dimensional Gaussian sphere feature. Finally, the cloud service can input this target three-dimensional Gaussian sphere feature into a Gaussian splash model to obtain the predicted language features of the sample image.

[0182] Optionally, the cloud service can input the aforementioned 3D Gaussian sphere features of the target into the Gaussian splash model to obtain 3D grid features. This allows for convenient subsequent determination of the target object's 3D position information based on these grid features, providing traversal conditions for indoor robots to determine object positions.

[0183] In some embodiments, after acquiring the sample image, the cloud service can input the sample image into an image segmentation model to obtain an image segmentation result. This image segmentation result includes the set of pixels corresponding to each object in the sample image. Then, the cloud service can input the set of pixels corresponding to multiple objects in the sample image into a visual language model to obtain the true language features of the sample image. Subsequently, the cloud service can train a semantic prediction model based on the predicted language features and the true language features of the sample image to obtain a target semantic prediction model.

[0184] In another implementation, such as Figure 13 As shown, this is one possible implementation process for an indoor robot to train a target semantic prediction model based on a sequence of target images. Figure 13 As shown, the process specifically includes: an indoor robot acquiring a target image sequence 910. This target image sequence includes multiple sample images from different camera poses. Then, the indoor robot can obtain a target rendered image 920 and a 3D occupancy grid 930 based on the multiple sample images in the target image sequence and the corresponding camera pose for each sample image. Next, the indoor robot can train a pre-built semantic prediction model based on the target rendered image 920 and the 3D occupancy grid 930 to obtain a target semantic prediction model.

[0185] It is understood that the aforementioned target semantic prediction model is trained and deployed by an indoor robot to detect the position information of target objects. In other words, the indoor robot is the primary entity executing the training process of this target semantic prediction model. However, in some embodiments, to reduce the cost and improve the efficiency of model training, the aforementioned target semantic prediction model can be trained using cloud services. That is, after obtaining the target semantic prediction model, the cloud service can send it to the indoor robot, effectively deploying the target semantic prediction model within the robot. This enables timely detection of object positions, allowing users to quickly locate target objects and improving the user experience.

[0186] The following section provides a detailed explanation of the training process of the aforementioned target semantic prediction model, using cloud services as the execution entity. Specifically, for example... Figure 14 As shown, the training process of this target semantic prediction model may include S701 to S708:

[0187] S701, cloud service acquires target image sequence.

[0188] The target image sequence includes multiple sample images of the target 3D scene under different camera poses. That is, the 3D scene captured by these multiple sample images belongs to a portion of the target 3D scene, and each sample image corresponds to a camera pose, which represents the position and orientation of the camera that captured the sample image within the target 3D scene. The target 3D scene represents the environment in which the camera is located. In some embodiments, the camera pose can be determined based on a localization algorithm. For example, the localization algorithm may include simultaneous localization and mapping (SLAM) algorithms, perspective-n-points (PNP), etc.

[0189] It is understood that the aforementioned target image sequence is based on data captured by the camera of the indoor robot. That is, after the indoor robot's camera captures multiple sample images, these sample images can be sent to a cloud service so that the cloud service can train a target semantic prediction model based on these sample images. The number of sample images captured can be preset. Furthermore, the camera installed on the indoor robot is an RGB camera, meaning that the sample images captured by this RGB camera are RGB images.

[0190] S702, for each sample image in the target image sequence, the cloud service renders the initialized radiation field based on the sample image to obtain the rendered image and the language features of the rendered image.

[0191] Specifically, after acquiring the target image sequence, the cloud service can initialize the query parameters of the radiation field to obtain an initialized radiation field. These query parameters are randomly assigned by the cloud service; that is, they are randomly set. Then, for each sample image in the target image sequence, the cloud service can render the initialized radiation field based on that sample image, obtaining a rendered image and its language features. This rendered image is obtained by rendering based on the camera pose corresponding to the sample image within the initial 3D scene. The initial 3D scene is the 3D scene corresponding to the initialized radiation field.

[0192] For example, the radiation field mentioned above can be a neural radiance field (NeRF) or a three-dimensional Gaussian sphere, and there is no specific limitation.

[0193] In some embodiments, the cloud service can input the aforementioned sample image and its corresponding camera pose into an initialized radiation field to obtain a first radiation field. This first radiation field characterizes the color and linguistic features of each voxel in the 3D scene under the camera pose corresponding to the sample image. Subsequently, the cloud service can use differentiable rendering to render the first radiation field to obtain the rendered image and its linguistic features.

[0194] Optionally, if the above radiation field is a three-dimensional Gaussian sphere, the cloud service can also use the Gaussian splash model to render the above first radiation field to obtain a rendered image and the language features of the rendered image.

[0195] S703, the cloud service uses an image segmentation model to segment the sample image and obtain the image segmentation result. This image segmentation result includes the set of pixels corresponding to each object in the sample image.

[0196] In some embodiments, after obtaining the target image sequence, the cloud service can input sample images from the target image sequence into an image segmentation model to obtain image segmentation results. These image segmentation results include the set of pixels corresponding to each object in the sample images.

[0197] It can be understood that the above image segmentation model is a pre-trained image segmentation model.

[0198] S704, the cloud service uses a visual language model to extract features from the set of pixels corresponding to multiple objects in a sample image, thus obtaining the language features of the sample image.

[0199] Specifically, after obtaining the image segmentation results described above, the cloud service can input the set of pixels corresponding to multiple objects in the sample image into a visual language model to obtain the language features of the sample image. These language features are used to characterize the real object information corresponding to each pixel in the sample image.

[0200] It is understood that the above visual language model is a pre-trained visual language model.

[0201] In this embodiment, the cloud service uses image segmentation and visual language models to extract real language features from sample images as supervisory data for optimizing the radiation field. This means the cloud service can optimize the initialized radiation field based solely on sample images, thereby obtaining the target semantic prediction model. This not only avoids relying on manually labeled semantic tags during training, saving significant time and manpower costs and improving the training efficiency of the semantic prediction model, but also allows the supervisory data to be unrestricted by manually labeled semantic tags, enabling semantic prediction for any open vocabulary. This enhances the application flexibility and semantic prediction performance of the target semantic prediction model, providing a foundation for accurate subsequent detection of target objects.

[0202] It should be noted that the process of determining the language features of the rendered image and the process of determining the language features of the sample image can be executed simultaneously or sequentially, without any specific limitation. For example, the cloud service can execute step S702 simultaneously, as well as steps S703 to S704. Alternatively, the cloud service can execute steps S703 to S704 first, and then execute step S702.

[0203] S705, the cloud service determines the color loss value based on the rendered image and the sample image, and determines the feature loss value based on the language features of the rendered image and the sample image.

[0204] In some embodiments, after obtaining the rendered image, the cloud service can determine the color loss value under the camera pose corresponding to the sample image based on the rendered image and the sample image. This color loss value characterizes the color error between the rendered image and the sample image. Furthermore, the cloud service can determine the feature loss value under the camera pose corresponding to the sample image based on the language features of the rendered image and the sample image. This feature loss value characterizes the feature error between the language features of the rendered image and the language features of the sample image, that is, the error between the predicted language features and the true language features of the sample image.

[0205] S706, the cloud service optimizes the initialized radiation field based on the color loss value and the feature loss value to obtain the target radiation field.

[0206] Specifically, after obtaining the aforementioned color loss value and feature loss value, the cloud service can optimize the initialized radiation field based on these values ​​to obtain the target radiation field. This target radiation field is used to characterize the color and linguistic features of each voxel in the 3D scene. In other words, through this target radiation field, occluded portions of the image can be filled in, thereby improving the reconstruction accuracy of the 3D scene and maximizing the restoration of the 3D scene, thus laying a foundation for subsequent training of the semantic prediction model.

[0207] In some embodiments, after obtaining the color loss value and feature loss value of the camera pose corresponding to the sample image, the cloud service can adjust the parameters of the initialized radiation field according to the color loss value and feature loss value of the camera pose corresponding to multiple sample images, until the color loss value is less than the first preset loss value and the feature loss value is less than the second preset loss value, thereby obtaining the target radiation field.

[0208] It is understandable that by optimizing the color loss value and feature loss value under different camera poses, the occluded parts can be accurately filled in, so that the optimized target radiation field can generate rendering images from any viewpoint, improving the accuracy of image rendering, and thus providing a foundation for subsequent training of semantic prediction models.

[0209] In this embodiment, a 3D scene model adapted to the 3D scene is represented by the aforementioned target radiation field. That is, the 3D objects in the 3D scene are represented by the target radiation field, and this target radiation field is used to train the semantic prediction model. This avoids using large amounts of memory to store the complex scene being rendered, reducing memory consumption and thus improving the model's computational speed. Furthermore, by outputting color and language features in the 3D scene through the initialized radiation field, and obtaining the rendered image and its language features through differentiable rendering, the initialized radiation field is optimized to obtain the target radiation field, eliminating the need for 3D scene reconstruction based on 3D geometric data. This reduces the waste of significant resources due to reliance on expensive 3D geometric sensors to collect 3D geometric data, improving resource utilization.

[0210] The S707 cloud service processes the target radiation field based on the preset camera pose to obtain the target rendering image and three-dimensional raster features.

[0211] Specifically, after obtaining the target radiation field, the cloud service can perform differentiable rendering of the color of each voxel representing the target radiation field based on a preset camera pose, thus obtaining a target rendered image. This target rendered image is an image rendered based on the target 3D scene and the preset camera pose. In other words, the target rendered image can be a rendered image from any camera pose and viewpoint, not limited to the camera pose and viewpoint when acquiring the sample image. This allows for the detection of object positions from any angle, improving the accuracy of object position detection.

[0212] Furthermore, the cloud service can sample the linguistic features of each voxel representing the target's radiation field based on the preset camera pose, obtaining a 3D raster feature. In other words, this 3D raster feature can better represent the positional information of objects in a 3D scene, improving the accuracy of object position detection. Specifically, this 3D raster feature is used to represent the object information corresponding to each voxel in the 3D scene. This 3D raster feature includes multiple raster features, and the number of raster features is preset.

[0213] S708, the cloud service trains a pre-built semantic prediction model based on the target rendered image and 3D raster features to obtain the target semantic prediction model.

[0214] In some embodiments, after obtaining the aforementioned 3D raster features, the cloud service can use the target rendered image as input to the semantic prediction model and the 3D raster features as the actual output of the semantic prediction model to train the semantic prediction model, thereby obtaining the target semantic prediction model. In other words, the cloud service can train the target semantic prediction model using the target rendered image and the 3D raster features. This allows the target semantic prediction model to perform object detection for any open vocabulary, not limited to any fixed vocabulary in a fixed vocabulary list. This enables a more comprehensive search for target objects, reducing the occurrence of situations where the target object cannot be found due to the query being an open vocabulary, improving the accuracy of target object detection, and ultimately enhancing the user experience.

[0215] Specifically, after obtaining the aforementioned 3D raster features and the target rendered image, the cloud service can input the target rendered image into a pre-built semantic prediction model to obtain the predicted 3D raster features (or predicted raster features). Then, the cloud service can adjust the parameters of the pre-built semantic prediction model based on these predicted raster features and the 3D raster features to obtain the target semantic prediction model. This enables accurate training of the semantic prediction model, providing convenient conditions for subsequent accurate object location queries.

[0216] In other embodiments, the cloud service can obtain a semantic tag list. This semantic tag list includes semantic tags for at least one object, which represent object information, including category information and / or attribute information. It is understood that this semantic tag list is pre-set based on the user's query habits. For example, if a user frequently searches for "wooden comb," the semantic tag list may include semantic tags representing the category information as "wooden comb." Then, for each object's semantic tag in the semantic tag list, the cloud service can input the semantic tag into a visual language model to obtain the object's linguistic features. Subsequently, the cloud service can determine the target raster features matching the object from the aforementioned three-dimensional raster features based on the object's linguistic features. In other words, the target semantic prediction model is trained based on the semantic tags included in the semantic tag list, enabling it to perform object detection for object information corresponding to any semantic tag in the list. This means the target semantic prediction model can perform object detection for any fixed vocabulary, achieving accurate detection of target objects and thus improving the user experience.

[0217] Specifically, for each of the aforementioned 3D raster features, the cloud service can calculate the similarity between that raster feature and the linguistic features of the object. Then, the cloud service can use a similarity greater than a second preset similarity as the target raster feature. In other words, determining the target raster feature by calculating the similarity between features provides a foundation for subsequent accurate training of the semantic prediction model.

[0218] The above describes one training process for a target semantic prediction model. The following section will further detail a possible implementation process for training a target semantic prediction model using cloud services. For example... Figure 15 As shown, the training process can specifically include: the cloud service acquiring a target image sequence. This target image sequence includes multiple sample images from different camera poses. Then, for each sample image in the target image sequence, the cloud service can perform differentiable rendering of the initialized radiation field based on the sample image and its corresponding camera pose, obtaining a rendered image and its language features.

[0219] Furthermore, the cloud service can input the aforementioned sample images into an image segmentation model to obtain image segmentation results. These results include the set of pixels corresponding to each object in the sample image. Then, the cloud service can input the set of pixels corresponding to multiple objects in the sample image into a visual language model to obtain the language features of the sample image. Next, the cloud service can determine the color loss value based on the rendered image and the sample image, and determine the feature loss value based on the language features of the rendered image and the sample image. Finally, the cloud service can optimize the initialized radiation field based on the color loss value and the feature loss value to obtain the target radiation field.

[0220] Next, the cloud service can process the target radiation field based on the preset camera pose to obtain a target rendered image and 3D raster features. Then, the cloud service can obtain a semantic tag list. This list includes semantic tags for at least one object, which represent the object's information. For each object's semantic tag in the list, the cloud service can input it into a visual language model to obtain the object's linguistic features. Then, for each raster feature in the 3D raster features, the cloud service can calculate the similarity between the raster feature and the object's linguistic features. The cloud service can then select raster features with a similarity greater than a second preset similarity from the 3D raster features and use these as target raster features. Finally, the cloud service can train a semantic prediction model based on the target raster features and the target rendered image to obtain a target semantic prediction model.

[0221] In some embodiments, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0222] In some embodiments, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0223] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0224] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0227] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0228] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting the position of an object, characterized in that, Applied to electronic devices, the method includes: The electronic device acquires a target surround view image; wherein, the target surround view image is used to characterize the three-dimensional scene of the environment in which the electronic device is located; Upon receiving a query instruction, the electronic device uses a target semantic prediction model to perform semantic prediction on the target surround view image to obtain the semantic information of the three-dimensional scene; wherein, the query instruction includes object information of the object to be queried, the object information of the object to be queried includes attribute information and / or category information representing open words, and the semantic information includes object information corresponding to each voxel in the three-dimensional scene; The electronic device determines the position information of the object to be queried in the three-dimensional scene based on the semantic information of the three-dimensional scene and the object information of the object to be queried.

2. The method according to claim 1, characterized in that, The electronic device uses a target semantic prediction model to perform semantic prediction on the target surround view image to obtain the semantic information of the three-dimensional scene, including: The electronic device inputs the target surround view image into the target semantic prediction model to obtain three-dimensional raster features; wherein, the three-dimensional raster features include raster features of multiple raster cells, and the raster features are used to characterize the object information of voxels in the three-dimensional scene; The electronic device determines the position information of the object to be queried in the three-dimensional scene based on the semantic information of the three-dimensional scene and the object information of the object to be queried, including: The electronic device determines the position information of the object to be queried in the three-dimensional scene based on the three-dimensional grid features and the object information of the object to be queried.

3. The method according to claim 2, characterized in that, The electronic device determines the position information of the object to be queried in the three-dimensional scene based on the three-dimensional grid features and the object information of the object to be queried, including: The electronic device identifies the object information of the object to be queried and obtains the query text; The electronic device inputs the query text into a visual language model to obtain the linguistic features of the object to be queried; wherein, the linguistic features are used to characterize the object information of the object to be queried; The electronic device matches the linguistic features of the object to be queried with the three-dimensional grid features to obtain the location information of the object to be queried in the three-dimensional scene.

4. The method according to claim 3, characterized in that, The electronic device matches the linguistic features of the object to be queried with the three-dimensional raster features to obtain the location information of the object to be queried in the three-dimensional scene, including: For each grid feature in the three-dimensional grid features, the electronic device calculates the similarity between the grid features and the linguistic features of the object to be queried; The electronic device uses the grid features with a similarity greater than a preset similarity to form a three-dimensional mask of the object to be queried; wherein, the three-dimensional mask of the object to be queried is used to represent the position information of the object to be queried in the three-dimensional scene.

5. The method according to any one of claims 1-4, characterized in that, The target semantic prediction model is obtained by adjusting the parameters based on the predicted language features and the real language features of the sample images during iterative training. The predicted language features of the sample images are obtained by adjusting the parameters of the initialized three-dimensional Gaussian sphere based on the sample images, and the real language features of the sample images are obtained based on the sample images.

6. The method according to claim 5, characterized in that, The predicted language features of the sample image are obtained by rendering based on the target 3D Gaussian sphere features. The target 3D Gaussian sphere features are obtained by updating the 3D Gaussian sphere features using a self-attention network. The 3D Gaussian sphere features are obtained by fusing the initialized 3D Gaussian sphere and the 2D image features of the sample image. The 2D image features of the sample image are obtained by feature extraction based on the sample image.

7. The method according to claim 5 or 6, characterized in that, The true language features of the sample image are obtained by extracting features from the set of pixels corresponding to each object in the sample image using a visual language model. The set of pixels corresponding to each object in the sample image is obtained by segmenting the sample image using an image segmentation model.

8. A method for training a semantic prediction model, characterized in that, include: Obtain a sample database; wherein the sample database includes at least one sample image, and the sample image shows at least one object; For each sample image in the sample database, the parameters of the initialized three-dimensional Gaussian sphere are adjusted according to the sample image to obtain the target three-dimensional Gaussian sphere features of the sample image. Based on the Gaussian splash model, the three-dimensional Gaussian sphere features of the target are rendered to obtain the predicted language features of the sample image; Based on the predicted language features and the real language features of the sample images, a pre-constructed semantic prediction model is trained to obtain a target semantic prediction model; wherein, the real language features of the sample images are obtained based on the sample images, and the output of the target semantic prediction model is used to characterize the semantic information of the three-dimensional scene.

9. The method according to claim 8, characterized in that, The step of adjusting the parameters of the initialized three-dimensional Gaussian sphere according to the sample image to obtain the target three-dimensional Gaussian sphere features of the sample image includes: Based on the feature extraction network, feature extraction is performed on the sample image to obtain the two-dimensional image features of the sample image; wherein, the two-dimensional image features include the pixel features of each pixel in the sample image; The initialized three-dimensional Gaussian sphere and the two-dimensional image features of the sample image are fused to obtain the three-dimensional Gaussian sphere features; The three-dimensional Gaussian sphere features are updated using a self-attention network to obtain the target three-dimensional Gaussian sphere features of the sample image.

10. The method according to claim 9, characterized in that, The initialized three-dimensional Gaussian sphere comprises multiple Gaussian spheres. The process of fusing the initialized three-dimensional Gaussian spheres with the two-dimensional image features of the sample image to obtain three-dimensional Gaussian sphere features includes: Based on the intrinsic and extrinsic parameters of the camera, the pixel features corresponding to each Gaussian sphere are determined from the two-dimensional image features; wherein, the camera is the camera that captures the sample image; For each Gaussian sphere in the three-dimensional Gaussian sphere, the Gaussian sphere and its corresponding pixel features are input into a cross-attention network to obtain Gaussian sphere features; The Gaussian sphere features of the multiple Gaussian spheres are merged to obtain the three-dimensional Gaussian sphere features.

11. The method according to claim 9 or 10, characterized in that, The step of updating the 3D Gaussian sphere features based on a self-attention network to obtain the target 3D Gaussian sphere features of the sample image includes: If the sample database includes a sample image, or if the three-dimensional Gaussian sphere feature is a feature obtained when training the semantic prediction model for the first time, the three-dimensional Gaussian sphere feature is input into the self-attention network to obtain the target three-dimensional Gaussian sphere feature of the sample image. If the sample database includes at least two sample images, and the three-dimensional Gaussian sphere feature is not a feature obtained during the first training of the semantic prediction model, the three-dimensional Gaussian sphere feature and the historical three-dimensional Gaussian sphere feature are input into the self-attention network to obtain the target three-dimensional Gaussian sphere feature of the sample image; wherein, the historical three-dimensional Gaussian sphere feature is a feature obtained during the previous training of the semantic prediction model.

12. The method according to any one of claims 8-11, characterized in that, The method further includes: The sample image is input into an image segmentation model to obtain the image segmentation result of the sample image; wherein, the image segmentation result includes the set of pixels corresponding to each object in the sample image; The set of pixels corresponding to multiple objects in the sample image is input into the visual language model to obtain the true language features of the sample image.

13. The method according to any one of claims 8-12, characterized in that, After adjusting the parameters of the initialized three-dimensional Gaussian sphere according to the sample image to obtain the target three-dimensional Gaussian sphere features of the sample image, the method further includes: The target 3D Gaussian sphere features are input into the Gaussian splash model to obtain the 3D raster features of the sample image; wherein, the 3D raster features of the sample image are used to characterize the semantic information of the 3D scene to which the sample image belongs.

14. An electronic device, characterized in that, include: A camera, one or more processors, and one or more memories; the one or more processors are coupled to the camera and the one or more memories. The camera is used to acquire a target surround view image, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-7 or the method as described in any one of claims 8-13.

15. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7 or the method as described in any one of claims 8 to 13.

16. A computer program product, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7 or the method as described in any one of claims 8 to 13.

Images