Robot refined target positioning method and device based on condition multi-mode prompt

Abstract

The invention relates to the technical field of computer vision, and provides a robot refined target positioning method and device based on conditional multi-mode prompt.The method comprises the steps that images and texts are subjected to multiple times of cross coding, and target visual features and target language features are obtained; in each cross coding, determining a first prompt guide according to the ith visual feature, and obtaining an (i + 1) th language feature in combination with the ith visual feature; determining a second prompt guide according to the ith language feature, and obtaining an (i + 1) th visual feature; and performing cross attention calculation on the mapped visual features and the mapped language features to obtain new visual features and new language features for the robot to adjust the motion posture. According to the method, the advantages of early and late fusion are combined, the precise fine-grained target positioning of the robot can be realized according to the free-form language expression, and the refined target positioning efficiency and accuracy of the robot are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a robot refined target positioning method and device based on conditional multimodal prompting. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, visual positioning technology has made significant progress and is widely used in fields such as autonomous driving, intelligent manufacturing, robotics and drone navigation.

[0003] Through visual positioning technology, robots can understand and respond to user operations more naturally. This naturalness is not only reflected in the robot's accurate capture of user actions, but also in the robot's in-depth understanding of user intentions. For example, robots can use visual positioning technology to capture the patient's movement trajectory and provide personalized motion assistance based on the patient's rehabilitation needs.

[0004] In related technologies, a two-stage (including candidate generation and cross-modal matching) visual localization model is usually used to explore more effective cross-modal interactions, or to select the best matching candidates in an explainable reasoning method to achieve target detection and positioning. However, the two-stage visual localization model is a serial architecture, the model training efficiency is limited, and it is too dependent on the training effect of the candidate generation stage, resulting in low model robustness. When a one-stage visual localization method is used for target visual localization, the corresponding language features and visual features are first extracted through two independent encoders, and then the two types of features are cross-modally fused through an aggregation module. The fused feature representation capability is limited, and only coarse-grained target positioning can be achieved (such as output as a positioning box), which is difficult to meet the requirements for refined target positioning by the robot (such as output as pixel-level coordinates). Summary of the invention

[0005] The present invention provides a robot fine-grained target positioning method and device based on conditional multimodal prompts, which are used to solve the problems that when a two-stage visual positioning method used in the prior art is used to perform robot fine-grained target positioning, the training efficiency is limited and it depends on the training effect of the first stage, resulting in low positioning efficiency and low model robustness; when a one-stage visual positioning method is used, the cross-modal fusion information representation ability is low, resulting in the inability to meet the robot's fine-grained target positioning needs, thereby improving the robot's fine-grained target positioning efficiency and accuracy.

[0006] The present invention provides a robot refined target positioning method based on conditional multimodal prompting, comprising: The image and the text are cross-coded for multiple times to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained by taking the image as the initial input, and the first prompt guide and the i-th visual feature are language-coded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained by taking the text as the initial input, and the second prompt guide is visually coded to obtain the i+1-th visual feature; i is a positive integer greater than 0; The target visual features and the target language features are mapped to the same space, and the mapped visual features and the mapped language features are cross-attention calculated to obtain new visual features and new language features, so that the robot can adjust its movement posture according to the position decoding results under the condition of position decoding of the new visual features and the new language features.

[0007] According to a robot refined target positioning method based on conditional multimodal prompting provided by the present invention, the target visual features include the visual features output by each cross-coding, and the target language features include the language features output by the last cross-coding; Before mapping the target visual feature and the target language feature to the same space, the method further includes: The visual features outputted by each cross-coding are reshaped respectively, and the reshaped visual features are spliced ​​to obtain new target visual features.

[0008] According to a robot refined target positioning method based on conditional multimodal prompting provided by the present invention, the cross attention calculation is expressed by the following formula: ; in, For multi-head cross attention operation, It is a feed-forward network; is the mapped visual feature, is the mapped language feature, For the new visual feature, For the new language feature.

[0009] According to a robot refined target positioning method based on conditional multimodal prompting provided by the present invention, the visual encoding is implemented by the following steps: Visually encoding the image or visual features through a pre-trained Swin Transformer; The language encoding is achieved through the following steps: The text or language features are language encoded through the BERT model.

[0010] According to a robot refined target positioning method based on conditional multimodal prompts provided by the present invention, the first prompt guidance is determined according to the i-th visual feature obtained with the image as the initial input, including: Projecting the i-th visual feature into the language feature space based on the linear layer of the conditional multimodal prompt generator to obtain the first prompt guidance; The determining of the second prompt guide according to the i-th language feature obtained by taking the text as the initial input comprises: The i-th language feature is projected into a visual feature space based on the linear layer to obtain the second prompt guidance.

[0011] According to a robot refined target positioning method based on conditional multimodal prompting provided by the present invention, after obtaining the new visual features and the new language features, the method further includes: Based on the robot locator, pixel-level position decoding is performed according to the new visual features and the new language features to obtain the position decoding result; wherein the robot locator includes at least three upsampling decoders; The position decoding result is converted into a position according to a space coordinate system corresponding to the robot's mechanical arm to obtain positioning parameters, and the motion posture of the mechanical arm is adjusted according to the positioning parameters.

[0012] The present invention also provides a robot refined target positioning device based on conditional multi-modal prompting, comprising: The encoding module is used to perform multiple cross-coding on the image and the text respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained by taking the image as the initial input, and the first prompt guide and the i-th visual feature are language-encoded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained by taking the text as the initial input, and the second prompt guide is visually encoded to obtain the i+1-th visual feature; i is a positive integer greater than 0; The feature calculation module maps the target visual feature and the target language feature to the same space, and performs cross-attention calculation on the mapped visual feature and the mapped language feature to obtain new visual feature and new language feature, so that the robot can adjust its movement posture according to the position decoding results under the condition of position decoding of the new visual feature and the new language feature.

[0013] According to a robot refined target positioning device based on conditional multimodal prompting provided by the present invention, the device also includes: A decoding module, configured to, after obtaining the new visual features and the new language features, perform pixel-level position decoding based on the new visual features and the new language features based on a robot locator to obtain the position decoding result; wherein the robot locator includes at least three upsampling decoders; The motion adjustment module performs position conversion on the position decoding result according to the space coordinate system corresponding to the robot's mechanical arm to obtain positioning parameters, and adjusts the motion posture of the mechanical arm according to the positioning parameters.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for refined target positioning of a robot based on conditional multimodal prompting as described in any one of the above-mentioned methods.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a robot refined target positioning method based on conditional multimodal prompting as described in any one of the above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned robot refined target positioning methods based on conditional multimodal prompts.

[0017] The method and device for robot fine-grained target positioning based on conditional multimodal cues provided by the present invention obtain target visual features and target language features by cross-encoding images and texts based on language encoding and visual encoding multiple times, and obtain fine-grained target positioning results after cross-attention calculation and decoding of the above two types of features, thereby completing the adjustment of the robot's motion posture. It combines the advantages of early and late fusion, can realize the robot's precise and fine-grained target positioning according to free-form language expression, and improves the efficiency and accuracy of the robot's fine-grained target positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is one of the flow charts of the robot refined target positioning method based on conditional multimodal prompting provided by the present invention.

[0020] Figure 2 This is the second flow chart of the robot refined target positioning method based on conditional multimodal prompting provided by the present invention.

[0021] Figure 3 This is the third flow chart of the robot refined target positioning method based on conditional multimodal prompting provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of a robot refined target positioning device based on conditional multimodal prompting provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figure 1-Figure 4 The present invention describes a method and device for robot refined target positioning based on conditional multimodal prompting.

[0026] Figure 1 This is one of the flow charts of the robot refined target positioning method based on conditional multimodal prompting provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110, cross-encoding the image and text multiple times respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained with the image as initial input, and the first prompt guide and the i-th visual feature are language encoded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained with the text as initial input, and the second prompt guide is visually encoded to obtain the i+1-th visual feature; i is a positive integer greater than 0.

[0027] In this step, the image and text can be data acquired in real time, for example, by capturing images of the surrounding environment (such as people, animals, tables and chairs, or other objects) through a camera sensor; collecting voice data in the surrounding environment through a microphone, and converting the voice data into text data through speech recognition technology.

[0028] Specifically, a robot is equipped with a processor for executing the above-mentioned robot refined target positioning method based on conditional multimodal prompts. The robot is also equipped with a camera and a camera gimbal composed of a damping adjustable buffer hinge, and the camera gimbal can adjust the pitch angle of the camera; the robot is also equipped with a microphone for obtaining voice data during human-computer interaction.

[0029] In this step, the image and text may also be historical data uploaded by the server or data sent by a remote person.

[0030] In this embodiment, the model used for visual encoding includes but is not limited to a convolutional neural network (CNN), an autoencoder (Autoencoder), a recurrent neural network (RNN), a Transformer model or a vision transformer (ViT) model, etc.

[0031] In this embodiment, the model used for language encoding includes but is not limited to a neural network language model (NNLM), a word embedding model, a multimodal language model, or a pre-trained language model.

[0032] In this step, by performing multiple cross-encoding on the image and text respectively, visually perceived language features and language-perceived visual features are generated, thereby achieving comprehensive cross-perceived early fusion, allowing for a deeper exploration of cross-modal information, and generating visually perceived language features can reduce the impact of noise in the original language expression.

[0033] In this step, the first prompt guide is a feature vector of a visual feature after linear transformation. The first prompt guide may include one or more feature vectors. One visual feature is outputted by visual encoding at one time, and one visual feature corresponds to one prompt guide.

[0034] Figure 2 This is the second flow chart of the robot refined target positioning method based on conditional multi-modal prompts provided by the present invention. Figure 2 In the embodiment shown, the image v0 is input to the visual encoder M1, the visual feature v1 is output, and the text Input to language encoder S1, output language features ; The conditional multimodal cue generator converts v1 into the first visual cue guide, and then combines the first visual cue guide with the language feature Input to language encoder S2, output language features , and so on, after passing through four language encoders, we can get language features with visual perception ( , and ); The conditional multimodal prompt generator will It is converted into a second prompt guidance about the language, and then the second prompt guidance about the language is input into the visual encoder M2, and the visual feature v2 is output. And so on, after passing through four visual encoders, the visual features (v2, v3 and v4) with language perception can be obtained.

[0035] In this embodiment, the target visual feature may be one or more of the above-mentioned visual features with language perception, and the target language feature may also be one or more of the above-mentioned visually perceived language features.

[0036] For example, the target visual features include v2, v3, and v4, and the target language features are .

[0037] Step 120: Map the target visual features and the target language features to the same space, and perform cross-attention calculations on the mapped visual features and the mapped language features to obtain new visual features and new language features, so that the robot can adjust its motion posture according to the position decoding results while decoding the new visual features and the new language features.

[0038] In this step, the features of the two modalities, the target visual features and the target language features, can be mapped into a shared semantic space, so that features of different modalities can be compared and interacted in the same space.

[0039] For example, the visual modality and the language modality can be aligned through the attention mechanism based on the Transformer model; a shared embedding layer can also be designed to convert visual features and language features into embedding vectors of the same dimension to ensure that the features of the two modalities are represented in the same space.

[0040] exist Figure 2 In the embodiment shown, the target visual feature V and target language features L Mapped to the same space, the mapped visual features are obtained respectively And the mapped language features , and then use the multi-head cross attention mechanism to and Perform cross-attention calculations to obtain new visual features and new language features ; It should be noted that the multi-head cross-attention mechanism allows the model to focus on different parts of the input information at the same time by using multiple attention heads in parallel. In the multi-head cross-attention mechanism, Q (Query), K (Key) and V (Value) are three important vectors used to calculate attention weights and generate outputs.

[0041] In this embodiment, after extracting features from the encoder, cross-modal interaction is further used to fuse cross-modal features; specifically, only visual features are used. , which are reshaped and concatenated to create the final visual input ,in, ; In addition, the final output of the language encoder is used as the language input , these inputs are projected into a common space and obtained and ;in, is the field of real numbers, is the dimension of visual features, is the number of feature channels, is the spatial dimension; The data length is , the number of channels of language features is The corresponding field of real numbers; The data length is , the number of feature channels is The corresponding field of real numbers.

[0042] In this embodiment, the cross attention calculation is expressed by the following formula: ; in, For multi-head cross attention operation, It is a feed-forward network; is the visual feature after mapping, is the language feature after mapping, For new visual features, For new language features.

[0043] In this embodiment, a locator is also installed on the robot, which finally outputs the above new visual features after prompting and guiding the cross-modal interaction. and new language features , and then through the locator according to and The robot position is decoded to obtain a pixel-level representation of the positioning result, namely the positioning coordinates, and then the center position of the target object is determined. The robotic arm grabs the target object according to the coordinates of the center position to complete the human-computer interaction process.

[0044] The robot fine-grained target positioning method based on conditional multimodal cues provided in an embodiment of the present invention obtains target visual features and target language features by cross-encoding images and texts based on language encoding and visual encoding multiple times, and obtains fine-grained target positioning results after cross-attention calculation and decoding of the above two types of features, thereby completing the adjustment of the robot's motion posture. It combines the advantages of early and late fusion, can realize the robot's precise and fine-grained target positioning according to free-form language expressions, and improves the efficiency and accuracy of the robot's fine-grained target positioning.

[0045] In some embodiments, the target visual features include the visual features output by each cross-coding, and the target language features include the language features output by the last cross-coding. Before mapping the target visual features and the target language features to the same space, the robot refined target positioning method based on conditional multimodal cues also includes: reshaping the visual features output by each cross-coding separately, and splicing the reshaped visual features to obtain new target visual features.

[0046] In this embodiment, the target visual features include v2, v3 and v4, which are visual features with language perception of different contents. The target language features are .

[0047] In this embodiment, before projecting the features into the same space, v2, v3 and v4 are reshaped respectively to unify the format of visual features of different shapes or dimensions, and then the reshaped visual features are spliced ​​to obtain a visual feature that includes rich visual information and language information.

[0048] The robot refined target positioning method based on conditional multimodal cues provided in an embodiment of the present invention improves the representation capability of the target visual features by reshaping the visual features output by each cross-coding respectively and splicing the reshaped visual features, thereby improving the accuracy of the refined target positioning results.

[0049] In some embodiments, visual encoding is achieved by the following steps: visually encoding the image or visual features through a pre-trained Swin Transformer.

[0050] In this embodiment, the pre-trained Swin Transformer adopts a hierarchical structure, can extract features at different scales, and has a strong representation learning ability; each visual encoder It is divided into four stages: The output of the visual stage is expressed as ,in Indicates the number of channels of the feature map, Represents spatial dimensions (same as above ). Each language encoder It is divided into four stages: The output of the language stage is expressed as ,in is the maximum length of the expression, The number of channels representing language features. Bidirectional early fusion is achieved through phased interaction between these features.

[0051] In this embodiment, visual encoding of an image by a pre-trained Swin Transformer is achieved by the following steps: (1) The image data to be tested is preprocessed by resizing and normalizing to ensure that the input image meets the input requirements of the pre-trained model.

[0052] (2) Select a pre-trained Swin Transformer model according to the task requirements (such as classification, detection, segmentation, etc.), input the pre-processed image into the pre-trained Swin Transformer model, and the model outputs visual feature maps. These feature maps contain the feature information of the image at different visual scales and positions.

[0053] In this embodiment, language encoding is achieved through the following steps: language encoding is performed on text or language features through a BERT model.

[0054] Specifically, the steps of language encoding text through the BERT model include: (1) Preprocess the text to be tested by performing operations such as word segmentation, removing stop words, removing punctuation, and lowercasing to ensure that the input text meets the input requirements of the BERT model; in addition, Chinese text usually needs to be segmented, and you can use the Chinese word segmentation tool provided by BERT or other third-party word segmentation tools.

[0055] (2) Select an appropriate BERT model version (such as BERT-base, BERT-large, etc.) according to the task requirements (such as text classification, named entity recognition, question-answering system, etc.). The model can be pre-trained on a large-scale corpus (such as Wikipedia, BooksCorpus, etc.); transform the preprocessed text as follows: add special tags (such as [CLS] and [SEP]), construct input sequences, calculate input masks and paragraph indexes, etc., to ensure that the text is converted into an input format acceptable to the BERT model. Input the processed input sequence into the BERT model, and the model will output a series of hidden states, namely language features. These language features contain feature information of the text in different positions and contexts.

[0056] The robot refined target positioning method based on conditional multimodal cues provided in an embodiment of the present invention can extract features containing rich visual information by performing visual encoding of images or visual features through a pre-trained Swin Transformer, and can extract features containing rich language information by performing language encoding of text or language features through a BERT model, thereby providing high-quality feature data for subsequent cross-modal fusion.

[0057] In some embodiments, determining a first prompt guidance based on an i-th visual feature obtained with an image as initial input includes: projecting the i-th visual feature to a language feature space based on a linear layer of a conditional multimodal prompt generator to obtain a first prompt guidance.

[0058] exist Figure 2 In the illustrated embodiment, visual cues are derived from visual features by a conditional cue generator to guide language feature generation in the next stage; these cues are projected into the language feature space through a linear layer, and the projected cues are connected with the language features to form the input of the subsequent stage of the language encoder; the conditional cue generator uses pyramid average pooling to generate the first cue guidance.

[0059] Determining the second prompt guidance according to the i-th language feature obtained by taking the text as the initial input includes: projecting the i-th language feature to the visual feature space based on the linear layer to obtain the second prompt guidance.

[0060] exist Figure 2 In the illustrated embodiment, a conditional hint generator is used to derive hints about language from language features to guide the generation of visual features in the next stage; these hints are projected into the visual feature space through a linear layer, and the projected hints form the input of the subsequent stage of the language encoder; the conditional hint generator uses pyramid average pooling to generate a second hint guide.

[0061] The robot refined target positioning method based on conditional multimodal prompts provided in an embodiment of the present invention projects the i-th visual feature to the language feature space based on the linear layer of the conditional multimodal prompt generator to obtain the first prompt guidance, and projects the i-th language feature to the visual feature space based on the linear layer to obtain the second prompt guidance, thereby realizing early fusion across perceptions and thus improving the accuracy of subsequent refined target positioning.

[0062] In some embodiments, after obtaining new visual features and new language features, the robot refined target positioning method based on conditional multimodal cues also includes: performing pixel-level position decoding based on the new visual features and new language features based on the robot locator to obtain a position decoding result; wherein the robot locator includes at least three upsampling decoders; converting the position decoding result according to the spatial coordinate system corresponding to the robot's mechanical arm to obtain positioning parameters, and adjusting the motion posture of the mechanical arm according to the positioning parameters.

[0063] In this embodiment, the object locator consists of three upsampling decoder modules, which can fully integrate multi-scale visual information and language information to obtain pixel-level refined prediction results.

[0064] exist Figure 2 In the embodiment shown, by using the new visual features and new language features ; Perform upsampling decoding to obtain pixel-level refined prediction results .

[0065] In this embodiment, the robot's processor processes the input image and text by executing the above-mentioned robot refined target positioning method based on conditional multimodal prompts, generates a refined positioning result of the target object, and determines the center position of the target object. Then, the position of the target object in the image coordinate system is converted into a three-dimensional position in the robotic arm coordinate system through a rotation matrix transformation. Finally, the robotic arm grasps the target object according to the three-dimensional coordinates to complete the service process for the person.

[0066] The robot refined target positioning method based on conditional multimodal prompts provided in an embodiment of the present invention obtains pixel-level interactive object positioning results and their specific positions by performing pixel-level position decoding based on new visual features and new language features based on the robot locator, thereby realizing flexible adjustment of the robot arm's motion posture and improving the robot's motion control efficiency and accuracy.

[0067] Figure 3 This is the third flow chart of the method for robot refined target positioning based on conditional multimodal prompting provided by the present invention. Figure 3 In the illustrated embodiment, voice data of the surrounding environment is received by the robot's microphone, and the voice data is converted into text by the voice recognition module, and image data of the surrounding environment is acquired by the camera; the text and image data are input into the robot's refined language-visual positioning model based on conditional multimodal prompts (obtained by training the robot's refined target positioning method based on conditional multimodal prompts) to generate a refined positioning result of the target object, and then the position of the target object is transformed into the coordinate system of the robotic arm through coordinate transformation, so that the robotic arm can perform related operations.

[0068] The following is a description of the robot fine-grained target positioning device based on conditional multimodal prompts provided by the present invention. The robot fine-grained target positioning device based on conditional multimodal prompts described below and the robot fine-grained target positioning method based on conditional multimodal prompts described above can be referenced to each other.

[0069] Figure 4 is a schematic diagram of the structure of a robot refined target positioning device based on conditional multi-modal prompting provided by the present invention, such as Figure 4 As shown, the robot refined target positioning device based on conditional multimodal prompting includes: an encoding module 410 and a feature calculation module 420.

[0070] The encoding module 410 is used to perform multiple cross-coding on the image and the text respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained by taking the image as the initial input, and the first prompt guide and the i-th visual feature are language-encoded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained by taking the text as the initial input, and the second prompt guide is visually encoded to obtain the i+1-th visual feature; i is a positive integer greater than 0; The feature calculation module 420 maps the target visual features and the target language features to the same space, and performs cross-attention calculation on the mapped visual features and the mapped language features to obtain new visual features and new language features, so that the robot can adjust its movement posture according to the position decoding results while decoding the new visual features and the new language features.

[0071] The robot fine-grained target positioning device based on conditional multimodal prompts provided in an embodiment of the present invention obtains target visual features and target language features by cross-encoding images and texts multiple times based on language encoding and visual encoding, and obtains fine-grained target positioning results after cross-attention calculation and decoding of the above two types of features, thereby completing the adjustment of the robot's motion posture. It combines the advantages of early and late fusion, can realize the robot's precise fine-grained target positioning according to free-form language expression, and improves the efficiency and accuracy of the robot's fine-grained target positioning.

[0072] In some embodiments, the robot refined target positioning device based on conditional multimodal cues also includes: a decoding module, which is used to perform pixel-level position decoding based on the robot locator according to the new visual features and the new language features after obtaining the new visual features and the new language features, so as to obtain a position decoding result; wherein the robot locator includes at least three upsampling decoders; a motion adjustment module, which converts the position decoding result according to the spatial coordinate system corresponding to the robot's mechanical arm, obtains positioning parameters, and adjusts the motion posture of the mechanical arm according to the positioning parameters.

[0073] The robot refined target positioning device based on conditional multimodal prompts provided in an embodiment of the present invention obtains pixel-level interactive object positioning results and their specific positions by performing pixel-level position decoding based on new visual features and new language features based on the robot locator, thereby realizing flexible adjustment of the robot arm's motion posture and improving the robot's motion control efficiency and accuracy.

[0074] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a robot refined target positioning method based on conditional multimodal prompts, the method comprising: performing multiple cross-coding on the image and the text respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained with the image as the initial input, and the first prompt guide and the i-th visual feature are language-encoded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained with the text as the initial input, and the second prompt guide is visually encoded to obtain the i+1-th visual feature; i is a positive integer greater than 0; the target visual features and the target language features are mapped to the same space, and the mapped visual features and the mapped language features are cross-attention calculated to obtain new visual features and new language features, so that the robot can adjust the motion posture according to the position decoding results under the condition that the new visual features and the new language features are positionally decoded.

[0075] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot refined target positioning method based on conditional multimodal prompts provided by the above methods, the method including: performing multiple cross-coding on the image and the text respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained with the image as the initial input, and the first prompt guide and the i-th visual feature are language encoded to obtain the i+1th language feature; a second prompt guide is determined according to the i-th language feature obtained with the text as the initial input, and the second prompt guide is visually encoded to obtain the i+1th visual feature; i is a positive integer greater than 0; the target visual features and the target language features are mapped to the same space, and the mapped visual features and the mapped language features are cross-attention calculated to obtain new visual features and new language features, so that the robot can adjust the motion posture according to the position decoding results under the condition of position decoding of the new visual features and the new language features.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the robot refined target positioning method based on conditional multimodal prompts provided by the above-mentioned methods, the method comprising: performing multiple cross-coding on the image and the text respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained with the image as the initial input, and the first prompt guide and the i-th visual feature are language-encoded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained with the text as the initial input, and the second prompt guide is visually encoded to obtain the i+1-th visual feature; i is a positive integer greater than 0; the target visual features and the target language features are mapped to the same space, and the mapped visual features and the mapped language features are cross-attention calculated to obtain new visual features and new language features, so that the robot can adjust its motion posture according to the position decoding results under the condition that the new visual features and the new language features are position-decoded.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot refined target positioning method based on conditional multimodal prompts, characterized in that: include: The image and the text are cross-coded for multiple times to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained by taking the image as the initial input, and the first prompt guide and the i-th visual feature are language-coded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained by taking the text as the initial input, and the second prompt guide is visually coded to obtain the i+1-th visual feature; i is a positive integer greater than 0; The target visual features and the target language features are mapped to the same space, and the mapped visual features and the mapped language features are cross-attention calculated to obtain new visual features and new language features, so that the robot can adjust its movement posture according to the position decoding results under the condition of position decoding of the new visual features and the new language features.

2. The method for robot refined target positioning based on conditional multimodal prompting according to claim 1, characterized in that: The target visual feature includes the visual feature of each cross-coding output, and the target language feature includes the language feature of the last cross-coding output; Before mapping the target visual feature and the target language feature to the same space, the method further includes: The visual features outputted by each cross-coding are reshaped respectively, and the reshaped visual features are spliced ​​to obtain new target visual features.

3. The method for robot refined target positioning based on conditional multimodal prompting according to claim 1, characterized in that: The cross attention calculation is expressed by the following formula: ； in, For multi-head cross attention operation, It is a feed-forward network; is the mapped visual feature, is the mapped language feature, For the new visual feature, For the new language feature.

4. The method for robot refined target positioning based on conditional multimodal prompting according to claim 1, characterized in that: The visual encoding is achieved by the following steps: Visually encoding the image or visual features through a pre-trained Swin Transformer; The language encoding is achieved through the following steps: The text or language features are language encoded through the BERT model.

5. The method for robot refined target positioning based on conditional multimodal prompting according to claim 1, characterized in that: The determining of the first prompt guide according to the i-th visual feature obtained by taking the image as the initial input comprises: Projecting the i-th visual feature into the language feature space based on the linear layer of the conditional multimodal prompt generator to obtain the first prompt guidance; The determining of the second prompt guide according to the i-th language feature obtained by taking the text as the initial input comprises: The i-th language feature is projected into a visual feature space based on the linear layer to obtain the second prompt guidance.

6. The method for robot refined target positioning based on conditional multimodal prompting according to claim 1, characterized in that: After obtaining the new visual features and the new language features, the method further includes: Based on the robot locator, pixel-level position decoding is performed according to the new visual features and the new language features to obtain the position decoding result; wherein the robot locator includes at least three upsampling decoders; The position decoding result is converted into a position according to a space coordinate system corresponding to the robot's mechanical arm to obtain positioning parameters, and the motion posture of the mechanical arm is adjusted according to the positioning parameters.

7. A robot refined target positioning device based on conditional multimodal prompts, characterized in that: include: The encoding module is used to perform multiple cross-coding on the image and the text respectively to obtain target visual features and target language features; wherein, in each cross-coding, a first prompt guide is determined according to the i-th visual feature obtained by taking the image as the initial input, and the first prompt guide and the i-th visual feature are language-encoded to obtain the i+1-th language feature; a second prompt guide is determined according to the i-th language feature obtained by taking the text as the initial input, and the second prompt guide is visually encoded to obtain the i+1-th visual feature; i is a positive integer greater than 0; The feature calculation module maps the target visual feature and the target language feature to the same space, and performs cross-attention calculation on the mapped visual feature and the mapped language feature to obtain new visual feature and new language feature, so that the robot can adjust its movement posture according to the position decoding results under the condition of position decoding of the new visual feature and the new language feature.

8. The robot refined target positioning device based on conditional multi-modal prompting according to claim 7 is characterized in that: The device also includes: A decoding module, configured to, after obtaining the new visual features and the new language features, perform pixel-level position decoding based on the new visual features and the new language features based on a robot locator to obtain the position decoding result; wherein the robot locator includes at least three upsampling decoders; The motion adjustment module performs position conversion on the position decoding result according to the space coordinate system corresponding to the robot's mechanical arm to obtain positioning parameters, and adjusts the motion posture of the mechanical arm according to the positioning parameters.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for refined target positioning of a robot based on conditional multimodal prompting as described in any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for robot refined target positioning based on conditional multimodal prompting as described in any one of claims 1 to 6 is implemented.

Images