Detection model training method, detection method, equipment, storage medium and product

Abstract

The invention discloses a detection model training method, a detection method, equipment, a storage medium and a product, and relates to the field of computer vision. The method comprises the following steps: constructing a sample data set; constructing an operable region detection model, wherein the operable region detection model comprises an image segmentation module, an image encoder, a text encoder, a feature learning module and an output module; the image segmentation module segments the 2D image to obtain image blocks; the image encoder performs feature extraction on the image blocks to obtain feature vectors; the text encoder performs feature extraction on the affordability label text to obtain a group of word vectors of the affordability label; a feature learning module learns an association relationship between a feature vector of each image block and a word vector of each accessibility label; the output module calculates the probability of the operable area according to the incidence relation; and training and testing the operable region detection model by using the sample data set. According to the invention, detection of an infinite number of operable areas in the 2D image can be realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an operable area detection model training method, detection method, device, storage medium and product. Background Art

[0002] With the rapid development of intelligent technology, robotics has been widely used in all walks of life. When a robot performs an action based on a voice command, it needs to first detect the operable area of ​​the action, then operate the operable area, and perform the corresponding action. For example, if the voice command is to pour tea, the robot needs to first detect the handle of the teapot (the operable area), then hold the handle and perform the action of pouring tea.

[0003] Actionable region detection is a challenging problem in various types of robotic applications. Traditional actionable region detection methods are limited to a set of predefined affordance labels, which may limit the adaptability of intelligent robots in complex and dynamic environments. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present application provides a detection model training method, detection method, device, storage medium and product, which can detect an unlimited number of operable areas in a 2D image and realize zero-sample detection of operable areas.

[0005] In a first aspect, the present application provides a method for training an operable area detection model, comprising: Constructing a sample data set; wherein each sample in the sample data set includes a 2D image and an availability label text, and each availability label in the availability label text corresponds to an operable area in the 2D image; Constructing an operable area detection model; wherein the operable area detection model includes an image segmentation module, an image encoder, a text encoder, a feature learning module and an output module; the image segmentation module is used to segment the 2D image to obtain image blocks; the image encoder is used to extract features from each image block to obtain a feature vector of the corresponding image block; the text encoder is used to extract features from the availability label text to obtain a set of word vectors of the availability label; the feature learning module is used to learn the association between the feature vector of each image block and the word vector of each availability label; the output module is used to calculate the probability of the operable area according to the association; The sample data set is used to train and test the operable area detection model.

[0006] Furthermore, the image encoder selects the SigLIP model; and the text encoder selects the ViT-B / 32 text encoder in the CLIP model.

[0007] Furthermore, in the feature learning module, the calculation formula for the association relationship between the feature vector of each image block and the word vector of each availability label is: ; in, Indicates The feature vector of the image block is The correlation between the word vectors of affordance labels, Indicates The feature vector of the image block, Indicates The word vectors of affordance labels, represents the norm, Indicates transpose.

[0008] Furthermore, in the output module, the probability calculation formula of the operable area is: ; in, Indicates Availability labels corresponding to The probability that the image block is an operable area, Indicates The feature vector of the image block is The correlation between the word vectors of affordance labels, represents the number of affordance tags, represents a learnable parameter.

[0009] Furthermore, during the training phase, the parameters of the text encoder remain unchanged, and only the parameters of the image encoder and the output module are trained.

[0010] Furthermore, in the training phase, the negative log-likelihood loss function is used to calculate the loss value. The specific formula is: ; in, represents the loss value, represents the number of image blocks, Indicates Availability labels corresponding to The probability that an image block is an operable area.

[0011] In a second aspect, the present application provides a method for detecting an operable area, comprising: Calling an operable area detection model; wherein the operable area detection model is obtained by using the operable area detection model training method as described in the first aspect; Get the image and text instructions to be detected; According to the text instruction, the image to be detected is detected using the operable area detection model to obtain the operable area of ​​the image to be detected under the text instruction.

[0012] In a third aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program / instructions stored on the memory, wherein the processor executes the computer program / instructions to implement the steps in the operational area detection model training method as described in the first aspect, or the steps in the operational area detection method as described in the second aspect.

[0013] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps in the operational area detection model training method as described in the first aspect, or the steps in the operational area detection method as described in the second aspect.

[0014] In a fifth aspect, the present application also provides a computer program product, comprising a computer program / instructions, which, when executed by a processor, implement the steps in the operational area detection model training method as described in the first aspect, or the steps in the operational area detection method as described in the second aspect.

[0015] The beneficial effects of this application are: The present application detects the operable area of ​​a 2D image under an availability label by learning the association between image blocks and availability labels, which can realize the detection of an unlimited number of operable areas in a 2D image, achieve zero-sample detection of operable areas, and realize the detection of arbitrary operable areas without restricting the robot application to a framework with a fixed set of availability labels.

[0016] This application is applied to intelligent robots and can accurately output the robot's operable area according to instructions, thereby improving the robot's manipulation accuracy, realizing perfect interaction between user instructions and the robot, improving the efficiency of human-machine collaboration, improving the robot's perception of the surrounding environment and objects, and improving the robot's adaptability in complex and dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flow chart of the method for training an operable area detection model in an embodiment of the present application; Figure 2 This is a diagram of the operational area detection model architecture in the embodiment of the present application; Figure 3 This is a flow chart of the method for detecting an operable area in an embodiment of the present application; Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following is a clear and complete description of the technical solutions in this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Embodiment 1 Figure 1 The flowchart of the method for training the operable area detection model provided in this application is shown. Figure 1 As shown, in step S11, a sample data set is constructed.

[0022] In order to train and test the operable area detection model, a sample data set needs to be constructed. Each sample in the sample data set includes a 2D image and an availability label text. Each availability label in the availability label text corresponds to an operable area in the 2D image. For example, the 2D image is a teapot image, and the availability label text is pick, grab, hold, etc. Then the availability label-pick corresponds to the teapot lid in the teapot image, the availability label-grab corresponds to the handle in the teapot image, and the availability label-hold corresponds to the handle in the teapot image.

[0023] Next, go to step S12. In step S12, construct an operable area detection model. Figure 2As shown in FIG. 1 , the operable region detection model includes an image segmentation module, an image encoder, a text encoder, a feature learning module, and an output module. The image segmentation module is used to segment the 2D image to obtain image blocks, wherein the first i The image block is represented as Patchi; the image encoder is used to extract features from each image block to obtain the feature vector of the corresponding image block, where i The feature vector of an image block is expressed as ; The text encoder is used to extract features from the affordance label text to obtain a set of word vectors for affordance labels, where j The word vector of the affordance label is represented as ; The feature learning module is used to learn the association between the feature vector of each image block and the word vector of each availability label; the output module is used to calculate the probability of the operable area based on the association, thereby predicting the operable area of ​​the object in the 2D image under the availability label.

[0024] The image encoder is used to extract image features from the image block and generate a feature vector. In this embodiment, the SigLIP (Sigmoid Loss for Language Image Pre-Training) model is selected as the image encoder to extract image features. The 2D image is divided into n image blocks, and the image encoder generates n feature vectors: , Represents the field of real numbers.

[0025] The text encoder is used to extract text features from text and generate word vectors. In this embodiment, the ViT-B / 32 text encoder in the CLIP (Contrastive Language-Image Pre-training) model is selected as the text encoder of the model. For an availability label text, the text encoder generates m word vectors: .

[0026] In order to detect the operable area under any text instruction, it is necessary to learn the association between image features and potential affordance labels. In this embodiment, the feature learning module uses the cosine similarity function to associate the image features with the text features of the affordance label. The specific calculation formula is: (1) in, Indicates The feature vector of the image block is The correlation between the word vectors of affordance labels, Indicates The feature vector of the image block, Indicates The word vectors of affordance labels, denotes norm, superscript Indicates transpose.

[0027] The output module calculates the probability that the image block corresponding to the availability label is an operable area according to each association relationship, and determines whether the image block is an operable area under the availability label according to the probability. In this embodiment, the probability calculation formula of the operable area is: (2) in, Indicates Availability labels corresponding to The probability that the image block is an operable area, Indicates The feature vector of the image block is The correlation between the word vectors of affordance labels, represents the learnable parameters, Represents the number of affordance tags (i.e., the number of word vectors generated by the text encoder).

[0028] Next, go to step S13. In step S13, the operable area detection model in step S12 is trained and tested using the sample data set in step S11.

[0029] During the training and testing phases, the parameters of the text encoder remain unchanged, and only the parameters of the image encoder and output module are trained. That is, during the training and testing phases, the text encoder is frozen, and only the weight parameters of the image encoder and output module are updated.

[0030] In the training phase, the training goal is to maximize the correlation between the feature vector of the matching image patch and the word vector of the affordance label. Therefore, it can be achieved by optimizing the weighted negative log-likelihood loss of the softmax output of each image block of the entire 2D image. The specific formula is: (3) in, represents the loss value, represents the number of image blocks, Indicates Availability labels corresponding to The probability that an image block is an operable area.

[0031] Embodiment 2 Figure 3 The flowchart of the operable area detection method provided by the present application is shown in FIG. Figure 3As shown, at step S21, the operable area detection model is called, wherein the operable area detection model is trained using the operable area detection model training method in Example 1 of the present application.

[0032] Next, go to step S22. In step S22, obtain the image to be detected and the text instruction. For an intelligent robot, the image to be detected can be acquired by a visual sensor installed on the intelligent robot; the text instruction can be obtained by converting the user's voice instruction into text.

[0033] Next, go to step S23. In step S23, according to the text instruction, the operable area detection model is used to detect the image to be detected, and the operable area of ​​the image to be detected under the text instruction is obtained.

[0034] That is, the image to be detected is input into the image segmentation module of the operable area detection model, the image to be detected is segmented into multiple image blocks by the image segmentation module, and then the image encoder is used to extract features of each image block to obtain a set of feature vectors; the text instruction is input into the text encoder of the operable area detection model, and the text encoder is used to extract features of the text instruction to obtain word vectors; the feature learning module calculates the correlation between each feature vector and the word vector, and the output module calculates the probability that each image block is an operable area under the text instruction according to the correlation, and selects the image block corresponding to the maximum probability as the operable area, thereby achieving the purpose of detecting the required availability from the 2D image.

[0035] Based on the operable area detection model, the present application does not require pre-defined availability labels when detecting operable areas, can detect an unlimited number of operable areas in an image, and realizes zero-sample detection of operable areas, greatly improving the adaptability of intelligent robots in complex and dynamic environments.

[0036] Embodiment 3 The present application also provides an electronic device, such as Figure 4 As shown, the electronic device includes: a memory, a processor and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the steps in the operable area detection model training method of Example 1 of the present application, or the steps in the operable area detection method of Example 2 of the present application.

[0037] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) or the programs and / or data loaded from the storage part into the random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM, and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0038] The processor and memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0039] Although not shown, an embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the operational area detection model training method of embodiment one of the present application, or the steps in the operational area detection method of embodiment two of the present application are implemented.

[0040] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0041] Although not shown, an embodiment of the present application also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, implement the steps in the operational area detection model training method of embodiment one of the present application, or the steps in the operational area detection method of embodiment two of the present application.

[0042] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application. Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training an operable area detection model, characterized in that: The training method comprises: Constructing a sample data set; wherein each sample in the sample data set includes a 2D image and an availability label text, and each availability label in the availability label text corresponds to an operable area in the 2D image; Constructing an operable area detection model; wherein the operable area detection model includes an image segmentation module, an image encoder, a text encoder, a feature learning module and an output module; the image segmentation module is used to segment the 2D image to obtain image blocks; the image encoder is used to extract features from each image block to obtain a feature vector of the corresponding image block; the text encoder is used to extract features from the availability label text to obtain a set of word vectors of the availability label; the feature learning module is used to learn the association between the feature vector of each image block and the word vector of each availability label; the output module is used to calculate the probability of the operable area according to the association; The sample data set is used to train and test the operable area detection model.

2. The method for training an operable area detection model according to claim 1, characterized in that: The image encoder uses the SigLIP model; the text encoder uses the ViT-B / 32 text encoder in the CLIP model.

3. The method for training an operable area detection model according to claim 1, characterized in that: In the feature learning module, the calculation formula for the association between the feature vector of each image block and the word vector of each availability label is: ； in, Indicates The feature vector of the image block is The correlation between the word vectors of affordance labels, Indicates The feature vector of the image block, Indicates word vectors for affordance labels, " " represents the norm, Indicates transpose.

4. The method for training an operable area detection model according to claim 1, characterized in that: In the output module, the probability calculation formula of the operable area is: ； in, Indicates Availability labels corresponding to The probability that the image block is an operable area, Indicates The feature vector of the image block is The correlation between the word vectors of affordance labels, represents the number of affordance tags, represents a learnable parameter.

5. The method for training an operable area detection model according to claim 1, characterized in that: During the training and testing phases, the parameters of the text encoder remain unchanged, and only the parameters of the image encoder and output module are trained.

6. The method for training an operable area detection model according to any one of claims 1 to 5, characterized in that: During the training phase, the negative log-likelihood loss function is used to calculate the loss value. The specific formula is: ； in, represents the loss value, represents the number of image blocks, Indicates Availability labels corresponding to The probability that an image block is an operable area.

7. A method for detecting an operable area, characterized in that: The detection method comprises: Calling an operable area detection model; wherein the operable area detection model is obtained by using the operable area detection model training method according to any one of claims 1 to 6; Get the image and text instructions to be detected; According to the text instruction, the image to be detected is detected using the operable area detection model to obtain the operable area of ​​the image to be detected under the text instruction.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the steps in the operable area detection model training method as described in any one of claims 1 to 6, or the steps in the operable area detection method as described in claim 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps in the operable area detection model training method as described in any one of claims 1 to 6, or the steps in the operable area detection method as described in claim 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the operable area detection model training method as described in any one of claims 1 to 6, or the steps in the operable area detection method as described in claim 7 are implemented.

Images