Cross-granularity target detection method and device, and electronic device

Abstract

The application provides a cross-granularity target detection method and device and electronic equipment, which can be used in the field of artificial intelligence technology. The method comprises: acquiring a target image and target task information; generating a detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained in sequence; and gradually identifying the position of each detection target in the target image in sequence according to the sequence of each detection target in the detection target sequence by a recursive method. The application realizes layer-by-layer accurate identification from coarse-granularity targets to fine-granularity targets, breaks through the limitations of current target detection technology in a robot perception system, and especially solves the challenges in processing fine-granularity target identification and accurate positioning in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a cross-granularity target detection method and device and electronic equipment. BACKGROUND

[0002] Robot perception technology has made significant progress in recent years, greatly promoting the development of many fields such as industrial manufacturing, medical health, and human-machine collaboration. However, current robot detection technology mainly focuses on coarse-grained target recognition, that is, identifying and positioning the overall object, and shows obvious shortcomings when dealing with tasks that require fine-grained understanding. In scenarios such as fine assembly, complex human-computer interaction, and flexible object manipulation, robots not only need to understand the overall structure of the target object, but also need to accurately identify and locate the local details and structural relationships of the object to complete high-precision tasks. This demand poses higher challenges to the accuracy, generalization ability, and context understanding of existing perception technology. SUMMARY

[0003] To solve the problems in the prior art, the embodiments of the present application provide a cross-granularity target detection method, device and electronic equipment, which can at least partially solve the problems in the prior art.

[0004] According to a first aspect of the present application, a cross-granularity target detection method is provided. The method can include: obtaining a target image and target task information; generating a detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained in the order; and identifying the position of each detection target in the target image in the order of the detection target sequence by a recursive method.

[0005] In some embodiments, the generating a detection target sequence according to the target image and the target task information includes: generating a detection target sequence according to the target image and the target task information by a target visual language model, wherein the target visual language model is obtained by training a preset visual language model based on first annotation data, the first annotation data includes images and task information used as model input, and further includes a detection target sequence used as a label.

[0006] In some embodiments, the target visual language model generates a detection target sequence from the target image and the target task information, including: performing first visual encoding on the target image to generate a first visual feature vector; performing first language encoding on the target task information to generate a first language feature vector; fusing the first visual feature vector and the first language feature vector using a first cross-modal attention mechanism to obtain a first joint feature representation; and performing language decoding on the first joint feature representation to obtain the detection target sequence.

[0007] In some embodiments, in the detection target sequence, each detection target is presented in a textual description.

[0008] In some embodiments, the positions of each detection target in the target image are identified in a recursive manner according to the order of the detection targets in the detection target sequence, including: inputting the target image and the textual description of the first detection target in the detection target sequence into a target detector to obtain the position information of the first detection target in the target image output by the target detector; cropping the target image according to the position information of the first detection target in the target image to obtain a first sub-image containing the first detection target; inputting the first sub-image and the textual description of the second detection target in the detection target sequence into the target detector to obtain the position information of the second detection target in the first sub-image output by the target detector; cropping the first sub-image according to the position information of the second detection target in the first sub-image to obtain a second sub-image containing the second detection target; and so on until the position information of the last detection target in the detection target sequence in the Nth sub-image is obtained, where N is a positive integer greater than 2.

[0009] In some embodiments, the target detector processes the textual description and the image of each input detection target as follows: performing second visual encoding on the image to generate a second visual feature vector; performing second language encoding on the textual description of the detection target to generate a second language feature vector; fusing the second visual feature vector and the second language feature vector using a second cross-modal attention mechanism to obtain a second joint feature representation; and determining the position of the detection target in the image according to the second joint feature representation.

[0010] In some embodiments, before inputting the target image and the literal description of the first detection target in the detection target sequence into the target detector, the method further comprises: obtaining second annotation data, wherein the second annotation data comprises an image and a literal description of a detection target as model input, and further comprises position information as a label, the position information satisfying a target precision requirement; training a preset detector according to the second annotation data to obtain the target detector.

[0011] In some embodiments, the method further comprises: according to the position information of each identified detection target, performing motion control on the mechanical arm to enable the mechanical arm to complete the target task.

[0012] According to a second aspect of the present application, a cross-granularity target detection apparatus is provided. The cross-granularity target detection apparatus comprises: an acquisition module configured to acquire a target image and target task information; a generation module configured to generate a detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained; and an identification module configured to identify the position of each detection target in the target image in a recursive manner according to the order of each detection target in the detection target sequence.

[0013] According to a third aspect of the present application, an electronic device is provided. The electronic device can comprise a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the cross-granularity target detection method according to any of the above embodiments.

[0014] According to a fourth aspect of the present application, a computer readable storage medium is provided. The computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of the cross-granularity target detection method according to any of the above embodiments.

[0015] According to a fifth aspect of the present application, a computer program product is provided. The computer program product comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer can perform the steps of the cross-granularity target detection method according to any of the above embodiments.

[0016] The cross-granularity target detection method, electronic device, computer-readable storage medium, and computer program product provided in this application embodiment achieve accurate identification of targets layer by layer from coarse-grained targets to fine-grained targets, solving the limitations of current target detection technology in robot perception systems, especially the challenges faced in handling fine-grained target identification and accurate positioning in complex environments. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate examples according to this application and, together with this description, serve to explain the principles of this application.

[0018] Figure 1 A flowchart illustrating an example of a cross-granularity target detection method of this application is shown.

[0019] Figure 2 The flowchart illustrating how the target visual language model of this application processes target images and target task information is shown.

[0020] Figure 3 The flowchart illustrating the process of obtaining the target visual language model by performing specialized training on a visual language model, as exemplified in this application, is shown.

[0021] Figure 4 A partial flowchart of an example cross-granularity target detection method of this application is shown.

[0022] Figure 5 This is a schematic diagram illustrating the processing flow of the target detector in this application for each input text description and image of the target.

[0023] Figure 6 This is a schematic diagram of the training process of an object detector provided in an example of this application.

[0024] Figure 7 This is a schematic diagram of the cross-granularity target detection system provided in this embodiment.

[0025] Figure 8 and Figure 9 These are schematic diagrams illustrating the process of target detection using a cross-granularity target detection method in one example of this application.

[0026] Figure 10 This is a structural block diagram of a cross-grain size target detection device provided in an embodiment of this application.

[0027] Figure 11 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] Detailed reference is made to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, wherein the same reference numerals in different drawings denote the same or similar elements unless otherwise stated. In the following description of exemplary embodiments, the described embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with several aspects of the invention as described in the appended claims.

[0029] The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this invention and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein is intended to represent and include any or all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various types of information, such information should not be limited by these terms. These terms are only used to distinguish the types of information. For example, without departing from the scope of the invention, first information may be referred to as second information, and similarly, second information may be referred to as first information. The term “if” as used herein may be understood, depending on the context, to mean “at the time,” “at this moment,” or “in response to a judgment.”

[0031] Traditional object detection methods typically focus only on single-granularity object recognition, making it difficult to handle precise localization tasks involving detailed object components. This results in insufficient perception accuracy and operational capabilities for robots performing complex operations. Therefore, this application aims to provide a cross-granularity object detection method that achieves accurate recognition layer by layer from coarse-grained to fine-grained targets, addressing the limitations of current object detection technologies in robot perception systems, particularly in handling the challenges of fine-grained target recognition and precise localization in complex environments.

[0032] The execution subject of the cross-granularity target detection method provided in this application includes, but is not limited to, a computer.

[0033] Figure 1 A flowchart illustrating a cross-granularity target detection method as exemplified in this application is shown. Figure 1 As shown, the cross-granularity target detection method provided in this application includes:

[0034] S1. Obtain target image and target task information;

[0035] In step S1, the target image is generally associated with the target task, which can be a detection task of one or more detection targets in the target image, or an operation task of one or more detection targets in the target image, and the target task information includes but is not limited to a text form. When the target task is to detect or operate an object with a physical structure, the detection target can specifically refer to an object, a component or an assembly of the object. For example, the target task is to "detect the handle of the refrigerator", and the target image contains a refrigerator (which can contain a part or the whole of the refrigerator); or the target task is to open the refrigerator, and the target image contains a refrigerator (which can contain a part or the whole of the refrigerator).

[0036] In step S2, a detection target sequence is generated according to the target image and the target task information, wherein each detection target in the target image is contained in the detection target sequence, and each detection target in the detection target sequence is sequentially contained.

[0037] In step S2, the target task can be understood by using artificial intelligence technology, and the visual features of the target image can be identified, and then a detection target sequence suitable for the target image and the target task information can be generated. For example, a language model (such as a visual model based on a Transformer architecture, a visual model based on a convolutional neural network (CNN), a language model based on a hybrid architecture, etc.) can be used to extract semantic features of the target task information, a visual model (such as a visual model based on a Transformer architecture, a visual model based on a convolutional neural network (CNN), etc.) can be used to extract visual features of the target image, and then the semantic features and the visual features can be fused, and a detection target sequence can be generated based on a language decoder according to the joint feature representation.

[0038] In the detection target sequence, each detection target can be presented in the form of a text description, and of course, it can also be presented in the form of a picture, code or symbol, etc., which is not limited in the embodiment.

[0039] In generating the detection target sequence, the target task and the target image are comprehensively considered to ensure that each detection target in the generated detection target sequence exists in the target image, and to complete the detection from coarse granularity to fine granularity, it is also necessary to ensure that each detection target in the detection target sequence is sequentially contained. It can be understood that the cross-granularity target detection method of the present application is to detect each detection target in the detection target sequence as a target.

[0040] S3, in the order of the detection targets in the detection target sequence, step by step identify the location of each detection target in the target image by a recursive method.

[0041] In step S3, specifically, for the first detection target in the detection target sequence, identify the location of the first detection target in the target image; according to the location of the first detection target, crop the target image to obtain a first sub-image containing the first detection target; identify the location of the second detection target in the detection target sequence in the first sub-image; according to the location of the second detection target, crop the first sub-image to obtain a second sub-image containing the second detection target; and so on, until the location of the last detection target in the detection target sequence is obtained, achieving accurate positioning of the target.

[0042] Specifically, the workflow of detection starts with coarse-grained target detection, first identifying the approximate type of the object, such as "door" or "cup", and generating preliminary detection cues. Combining the semantic information of the object with visual features, more detailed target detection cues are generated. These cues are not just a preliminary description of the object, but they can guide the system to further explore the fine-grained features of the object. For example, after identifying the coarse-grained target "door", the semantic relationship of the object is analyzed to generate detection cues for the fine-grained target "door handle", guiding subsequent detection to focus on the precise location of the door handle. Each step in this process is based on the results of the previous step, gradually refining the location and features of the target components, ensuring high-precision positioning of the target.

[0043] As can be seen, the cross-grained target detection method provided in the examples of the present application realizes higher-precision target positioning from coarse-grained target detection to fine-grained target identification.

[0044] Applying this method to the field of robotics can significantly improve the perception ability of robotic systems, especially in tasks that require precise control and grasping, such as grasping door handles, operating bottle caps, etc. This enables robots to achieve more efficient and accurate target recognition and operation in complex environments, not only improving the adaptability of robots in tasks, but also increasing their execution precision.

[0045] In some embodiments, the above step S2 can include generating a detection chain by a target visual language model according to the target image and the target task information. Specifically, the target visual language model (VLM, Visual Language Model) can not only identify the visual features of the detection target in the image, but also generate more detailed detection prompts by analyzing the semantic information of the detection target, further guiding the positioning of the fine-grained detection target. The target visual language model provides multi-level semantic and visual support for target detection, which can help understand and identify the position and features of fine-grained detection targets. For example, after identifying the "door", the target visual language model generates a detection prompt for the "door handle" component, significantly improving the accuracy and robustness of target detection.

[0046] Figure 2 A flowchart of the target visual language model of the present application example processing the target image and the target task information is shown. As shown in Figure 2 The target visual language model 01 can include a first visual encoder 11, a first language encoder 12, a first cross-modal feature fusion module 13, and a language decoder 14. The specific process of the target visual language model 01 generating a detection target sequence according to the target image and the target task information is as follows:

[0047] Step one, the first visual encoder 11 encodes the target image x to generate a first visual feature vector φ(x), which can be a high-dimensional visual feature vector.

[0048] Step two, the first language encoder 12 generates a first language feature vector ψ(S) according to the target task information S, which can be an embedding vector.

[0049] Step three, the first cross-modal feature fusion module 13 fuses the first visual feature vector φ(x) and the first language feature vector ψ(S) through a cross-modal attention mechanism to generate a first joint feature representation ξ(x, S).

[0050] Step four, the language decoder 14 decodes the first joint feature representation ξ(x, S) to generate a detection target sequence C = {c1, c2, …, c n}.

[0051] The order of the above steps one and two is not sequential, and they can be executed simultaneously. The formula representation of the above process is:

[0052] φ(x) = VisionEncoder(x), ψ(S) = LanguageEncoder(S)

[0053] ξ(x, S) = CrossModalFusion(φ(x), ψ(S))

[0054] C = LanguageDecoder(ξ(x, S)) = {c1, c2, …, c n}

[0055] In the formula, C is the generated detection target sequence, c i represents the i-th detection target in the detection target sequence, S is the natural language description (text description) of the target task, and the target image is x.

[0056] The target visual language model can be obtained by special training of a visual language model, further improving the adaptability to the detection target sequence generation task. Figure 3 A flowchart of the present application example of training a visual language model to obtain the target visual language model is shown. As Figure 3 shown, the training process is as follows:

[0057] S01, obtaining a first annotation data set, wherein the first annotation data set includes at least two first annotation data, and each first annotation data includes an image, a task information and a detection target sequence;

[0058] In step S01, the first annotation data set is used in special training, that is, the first annotation data, and the detection target sequence of each first annotation data contains detection targets of multiple granularities, such as "table", "drawer", "handle", etc. The target visual language model trained by the annotated data can more accurately generate a detection target sequence containing detection targets of different granularities in the multi-modal reasoning process.

[0059] S02, training a preset visual language model according to each first annotation data in the first annotation data set to obtain a target visual language model.

[0060] In step S02, the preset visual language model can be any visual language model, and each first annotation data in the first annotation data set is used to recursively train the preset visual language model. In the training process, the loss function is used to constrain the matching degree between the detection target sequence C generated by the visual language model and the real annotated detection target sequence C GT (the detection target sequence in the first annotation data). For example, the loss function can be defined as:

[0061] L = CrossEntropy(C, C GT )

[0062] Through training, the visual language model can generate a multi-granularity target detection target sequence that meets the demand more accurately. It can be seen that, by combining the powerful multi-modal understanding ability of the visual language model and the special optimization of the labeled data, a detection target sequence containing multi-granularity targets can be efficiently generated, which provides accurate task guidance for subsequent target accurate positioning.

[0063] In the detection target sequence generated by the above step S20, each detection target in the detection target sequence can be presented in a textual description, and the textual description of each detection target can only include the name of the detection target, and can also include other information of the detection target. For example, the specific form of the detection target sequence can be a sequence of detection target names. For example, a detection target sequence can be "table", "drawer", and "handle".

[0064] As shown in FIG. 1, in some embodiments, the above step S3 can include: Figure 4

[0065] S31, inputting the target image and the textual description of the first detection target in the detection target sequence into a target detector to obtain position information of the first detection target in the target image output by the target detector;

[0066] S32, cropping the target image according to the position information of the first detection target in the target image to obtain a first sub-image containing the first detection target;

[0067] S33, inputting the first sub-image and the textual description of the second detection target in the detection target sequence into the target detector to obtain position information of the second detection target in the first sub-image output by the target detector;

[0068] S34, cropping the first sub-image according to the position information of the second detection target in the first sub-image to obtain a second sub-image containing the second detection target;

[0069] S35, and so on until the position information of the last detection target in the detection target sequence in the Nth sub-image is obtained, where N is a positive integer greater than 2.

[0070] ​Specifically, the target detector (Detector) can be flexibly adapted to different detection tasks through language prompts (Prompt). The target detector can be an open-vocabulary object detection model (Open-Vocabulary Object Detection, OVOD). According to the order of the detection targets in the detection target sequence, the target detector is called to identify the target image and its subgraph, recursively refine and detect the detection targets, and realize cross-granularity target detection.

[0071] Figure 5 is a schematic diagram of the processing flow of the text description and image of the detection target input by the target detector in an example of the present application. As shown in Figure 5 , the target detector 02 includes a second visual encoder 21, a second language encoder 22, a second cross-modal feature fusion module 23, and a detection head (Detection Head) 24. The target detector 02 can simultaneously encode the input image and the text description of the detection target. Specifically, the second visual encoder 21 performs second visual encoding on the input image to generate a second visual feature vector, and the second language encoder 22 performs second language encoding on the text description of the detection target to generate a second language feature vector. The second cross-modal feature fusion module 23 fuses the second visual feature vector and the second language feature vector using a second cross-modal attention mechanism to obtain a second joint feature representation. The detection head 24 determines the position and category of the detection target in the image according to the second joint feature representation, where the category can be the name of the detection target identified by the detection head. In this example, the target detector uses a unified detection head to generate the position and category of the detection target, supporting open-vocabulary detection tasks. The above process can be represented by the following formula:

[0072]

[0073] wherein, is the detection result generated by the target detector in the input image x i and the text description c i of the detection target.

[0074] When a specific task is required, the detector can be fine-tuned using a custom precision labeled dataset to obtain the above target detector, thereby enhancing the adaptability of the target detector to specific tasks. Figure 6 is a schematic diagram of the training process of the target detector provided by an example of the present application, as shown in Figure 6 , the target detector can be trained by performing the following steps:

[0075] S03, obtaining a second annotation dataset, wherein the second annotation dataset comprises at least two pieces of second annotation data, each piece of second annotation data comprising an image, a textual description of a detection target, and position information of the detection target in the image, the position information satisfying a target precision requirement;

[0076] In step S03, the second annotation dataset refers to a self-defined precision annotation dataset, and the second annotation data in the second annotation dataset are annotated according to a self-defined precision. Specifically, the position information of the detection target in the image in each piece of second annotation data is annotated according to a self-defined position precision.

[0077] S04, training a preset detector according to each piece of second annotation data in the second annotation dataset to obtain a target detector.

[0078] In step S04, the preset detector can be any open-vocabulary detection model. Each piece of second annotation data in the second annotation dataset is used to recursively fine-tune (or train) the preset detector, and the training process is optimized by the following loss function:

[0079]

[0080] In the formula, is a classification loss of a target category, is a regression loss of a target bounding box (a kind of position information).

[0081] In some embodiments, when the cross-granularity target detection method is applied to the field of robot technology, the method can further comprise: performing motion control on a robot arm according to the position information of each identified detection target, so that the robot arm completes the target task. Specifically, robot motion path planning can be performed according to the position information of each identified detection target. For example, for the position information of each detected detection target, the center point coordinates of the detection target with the finest granularity are calculated and sent to the motion planning module of the robot arm. The robot arm performs kinematic planning according to the center point coordinates to complete the target operation (such as grasping, moving, etc.).

[0082] For better understanding of the present application, the cross-granularity target detection system and method provided by the present application are described in detail below through a specific embodiment.

[0083] I. System architecture

[0084] Figure 7 FIG. 1 is a structural schematic diagram of the cross-granularity target detection system provided by the present embodiment. As shown in FIG. 1, the cross-granularity target detection system comprises a data preprocessing module 10, a target detection module 20, a target tracking module 30, and a target operation module 40. Figure 7As shown, the cross-granularity object detection system 03 of the present application includes the following main modules: a visual language model module 31, a chain of detection (CoD) module 32, and an object detector module 33.

[0085] 1. Visual language model module, which is responsible for generating a detection object sequence according to an input image and a detection task description. The visual language model module has strong multi-modal reasoning capability, including a visual encoder, a language encoder, a cross-modal interaction module, and a language decoder. The visual encoder is used to extract high-dimensional features of the image, the language encoder is used to extract language features of the detection task description, the cross-modal interaction module performs deep fusion of the visual features and the language features to understand the detection task and generate the detection object sequence, and the language decoder is used to generate a semantic detection object sequence. In the present system, the visual language model is further trained to improve its adaptability to the detection object sequence generation task. The special training uses a labeled detection object sequence dataset, which contains detection task samples of multi-granularity object components, such as "table", "drawer", "handle", etc. Through these labeled data, the model can more accurately generate detection object sequences containing different granularity components in the multi-modal reasoning process. The specific reasoning and training process is as follows:

[0086] (1) Image feature extraction: the input image x is processed by the visual encoder to generate a high-dimensional visual feature vector φ(x).

[0087] (2) Task description embedding: input the detection task description S into the language encoder to generate an embedding vector ψ(S).

[0088] (3) Cross-modal feature fusion: through a cross-modal attention mechanism, the visual feature φ(x) and the task description feature ψ(S) are fused to obtain a joint feature representation ξ(x, S).

[0089] (4) Detection object sequence generation: based on the joint feature representation ξ(x, S), the language decoder generates a detection object sequence C.

[0090] The formula representation of the above process is:

[0091] φ(x) = VisionEncoder(x), ψ(S) = LanguageEncoder(S)

[0092] ξ(x, S) = CrossModalFusion(φ(x), ψ(S))

[0093] C = LanguageDecoder(ξ(x, S)) = {c1, c2, …, c n}

[0094] where C is the generated detection target sequence, c i represents the i-th target component in the detection target sequence, and the task description S is the natural language description of the detection requirement, and the input image is x. In the special training process, using the labeled detection target sequence dataset, the matching degree between the detection target sequence C generated by the model and the real labeled detection target sequence C GT is constrained by the loss function. The loss function can be defined as:

[0095] L = CrossEntropy(C, C GT )

[0096] Through training, the model can more accurately generate a multi-granularity detection target sequence that meets the task requirements. This module combines the powerful multi-modal understanding ability of the visual language model and the special optimization of the labeled data, and can efficiently generate a detection target sequence containing multiple levels of target components, providing accurate task guidance for the subsequent detector module.

[0097] 2. Recursive detection module, chain recursive detection module is responsible for sequentially performing detection steps according to the detection target sequence generated by the visual language model module. The CoD module calls the target detector module to identify the image and its subgraph according to the order in the detection target sequence, recursively refining and detecting target components. Specifically, the CoD module realizes cross-granularity through the following steps:

[0098] (1) Detection step execution: sequentially perform each detection step according to the order in the detection target sequence.

[0099] (2) Target identification: call the target detector module to identify the target component of the current detection step.

[0100] (3) Recursive call: for the identified target region, perform the next detection step until the entire detection target sequence is completed.

[0101] 3. Target detector module (Detector), the target detector module (Detector) is responsible for identifying and positioning in the image or subgraph according to each target component in the detection target sequence. This module uses an open vocabulary detection model that can be flexibly adapted to different detection tasks through language prompts (Prompt). The working principle of the open vocabulary detection model is as follows:

[0102] (1) Multi-modal feature extraction: the open vocabulary detection model encodes the input image and the target component description simultaneously. The image generates a feature representation through a visual encoder, and the target component description generates an embedding vector through a language encoder.

[0103] (2) Cross-modal feature fusion: The open vocabulary detection model utilizes a cross-modal attention mechanism to deeply fuse image features and language features, thereby locating and classifying targets in a joint feature space.

[0104] (3) Target detection and localization: Based on the fused features, the open vocabulary detection model uses a unified detection head to generate the location and class of the target, supporting the detection task of open vocabulary.

[0105] The formula of the above process is:

[0106]

[0107] where, is the detection result generated by the open vocabulary detection model on the input subgraph x i and target component description c i , representing the location and class of the target region.

[0108] In this system, the open vocabulary detection model is fine-tuned by using a custom precision annotation dataset, thereby enhancing its adaptability to specific tasks. The fine-tuning process is optimized by the following loss function:

[0109]

[0110] where, is the classification loss of the target class, is the regression loss of the target bounding box.

[0111] II. Detailed workflow

[0112] The entire detection process can include the following steps:

[0113] 1. Detection target sequence generation: The system uses a visual language model (VLM) to generate a detection target sequence C = {c1, c2,..., cn} based on the input image x and the detection task description S. The visual language model extracts visual features from the input image and generates a target component sequence by combining language prompts through multi-modal feature fusion and context understanding capabilities.

[0114] The specific process of the visual language model generating the detection target sequence is as follows:

[0115] (1) The image x is extracted by the visual encoder to obtain high-dimensional features φ(x).

[0116] (2) The detection task description S is generated by the language encoder to obtain the embedding vector ψ(S).

[0117] (3) Through the cross-modal feature fusion module, the image features φ(x) and language features ψ(S) are combined to generate the joint features ξ(x, S).

[0118] (4) Based on the joint features, the detection target sequence C = {c1, c2,..., cn} is generated. The formula is expressed as:

[0119] C = VLM(x, S) = {c1, c2,..., cn}, ξ(x, S) = CrossModalFusion(φ(x), ψ(S))

[0120] 2. Cross-granularity target recognition: For each detection prompt ct in the detection target sequence, the system calls the target detector module (Detector) for recognition in the image or subgraph. The target detector outputs the position information and class label of the target region according to the detection prompt ct and the current image xt. The target detector can support multi-granularity target detection tasks through the fine-tuning of the open vocabulary detection and labeled dataset.

[0121] The detailed workflow of the target detector is as follows:

[0122] (1) Input the current image or subgraph xt and the detection prompt ct;

[0123] (2) The target detector extracts image features and language features of target descriptions, and performs feature fusion through cross-modal attention mechanism;

[0124] (3) Based on the fused multi-modal features, the target detector outputs the bounding box of the target region and the class label The target detector is fine-tuned through the following loss function:

[0125]

[0126] where, is the cross-entropy loss of the target class, is the bounding box regression loss.

[0127] The formula of the above process is expressed as:

[0128]

[0129] 3. Recursive call and refinement: The system recursively calls the target detector according to the next step c t+1 of the detection target sequence, and further refines the detected content d t until all components are recognized.

[0130] 4. Failure handling and recovery: If a step fails (e.g., the detector fails to recognize the target), the system attempts the following recovery measures: (a) adjust the camera orientation to re-take the image; (b) increase the light source to improve image quality; (c) if multiple attempts still fail to detect the target, stop operation and output a failure prompt to avoid false actions.

[0131] 5. Robot path planning: For each detected target region, the system calculates the coordinates of the object center point and sends it to the motion planning module of the robot arm. The robot arm performs kinematic planning based on the target position to complete the target operation (such as grabbing, moving).

[0132] III. Examples

[0133] 1. Detailed description of the cross-granularity target detection method provided by the present application through the cross-granularity detection of the handle on the table.

[0134] (1) Input image: the system receives an input image x containing a table.

[0135] (2) Detection target sequence generation: the VLM generates a detection target sequence C = {table, drawer, handle}.

[0136] (3) Cross-granularity detection: (a) detect "table" and extract table region sub-image x1; (b) detect "drawer" and extract drawer region sub-image x2; (c) detect "handle" and identify the target region. 4. Path planning: send the identified "handle" center point coordinates to the robot arm motion module to complete the grabbing task.

[0137] 2. Detailed description of the cross-granularity target detection method provided by the present application in combination with the diagram.

[0138] Figure 8 and Figure 9 are respectively the process diagrams of the target detection using the cross-granularity target detection method in an example of the present application. Figure 8 and Figure 9 demonstrate how the recursive detection mechanism gradually detects objects from coarse granularity to fine granularity. Specifically, as shown in Figure 8 , starting from detecting "refrigerator", the system identifies "refrigerator door" and "refrigerator handle" to guide the robot to accurately perform specific operation tasks. Similarly, as shown in Figure 9As shown, for plastic bags, the process gradually identifies the handle and the functional components required for grasping. As can be seen, the cross-granularity target detection system and method provided by the present application has strong autonomous reasoning capability and can gradually optimize different levels of targets, thereby significantly improving the accuracy and robustness of the robot's perception capability. That is, the present application can achieve layer-by-layer refinement of the target during detection, dynamically adjust the detection path, provide efficient visual and language support for the robot, and complete multi-granularity target detection and accurate positioning in complex tasks.

[0139] As can be seen, the cross-granularity target detection method and system provided by the present application has the following characteristics:

[0140] Guidance of visual language model (VLM): The present application combines a visual language model (VLM) with traditional target detection methods. During the detection process, the VLM not only provides visual features of the object, but also generates more detailed detection prompts by analyzing the semantic information of the object, further guiding the detector to locate fine-grained targets. The VLM provides multi-level semantic and visual support for target detection, which can help the system understand and recognize the location and features of fine-grained components. For example, after identifying "door", the VLM generates a detection prompt for the "door handle" component, guiding the detector to accurately locate and identify the detailed features. This technology significantly improves the accuracy and robustness of target detection.

[0141] Chain recursive detection (CoD): The present application proposes a chain recursive detection algorithm, which generates a sequence of detection targets to gradually optimize the target recognition path according to the hierarchical relationship of target components. The recognition result of each layer is used as the input of the next layer detection, thereby gradually refining the target detection. Chain recursive detection can gradually refine the positioning and recognition of target components according to different levels of detection tasks, ensuring layer-by-layer optimization from the whole to the details. This process greatly improves the perception accuracy of robots in complex tasks and can effectively deal with the detection and positioning of fine-grained targets.

[0142] Dynamic adjustment of detection path: The chain detection method in the present application dynamically adjusts the detection path according to the relationship between the overall structure of the object and the detailed components, to optimize the target recognition process. Dynamic adjustment of the detection path can adjust the focus according to actual needs, from coarse-grained overall targets to fine-grained component targets, ensuring that the robot can flexibly cope with different tasks in complex environments. For example, after detecting "door", the system will preferentially focus on the detailed component "door handle" according to semantic information, thereby improving the efficiency and accuracy of task execution.

[0143] Recursive Refinement Object Recognition: The present application uses a recursive refinement approach, in which the system recursively calls the detector based on the order of the detected target sequence, and further refines each identified target region until all target components of the entire detected target sequence are identified. Recursive refinement ensures that each layer of target detection is optimized based on the previous step, gradually improving the accuracy of target recognition. This technology effectively avoids the precision problem in traditional detection methods, especially when the robot performs high-precision operations, ensuring accurate positioning of each target component.

[0144] Therefore, the cross-granularity object detection method and system provided by the present application has the following significant advantages:

[0145] Automatic detection: By combining visual language models (VLM) with detectors, the present application realizes an automatic detection process without human intervention. This not only greatly reduces the cost of manual detection, but also improves the speed and accuracy of target detection. Traditional methods require a large amount of manual annotation and adjustment, while the present invention greatly reduces the need for human intervention through intelligent reasoning and adaptive optimization. The present application uses visual language models to automatically generate detection prompts and automatically identifies and locates target components through detection algorithms. Since VLM can combine visual information and semantic information of objects, the detection process becomes more intelligent, reducing manual operations and improving automation level.

[0146] Fine detection capability: The present application uses a detection target sequence and a recursive optimization mechanism to enable the system to generate fine-grained detection results, significantly improving the accuracy and detail recognition capability of target recognition. This is crucial for handling complex tasks such as accurately grasping and positioning object components (e.g. "door handle" or "bottle cap"). The recursive detection algorithm allows the detection process to be optimized step by step, from coarse-grained target recognition to fine-grained component positioning. Each layer of recognition results is input for recursive refinement, ensuring accurate recognition at the detail level and improving overall accuracy and detail recognition capability.

[0147] High consistency and efficiency: The present application ensures the consistency and efficiency of the detection results through recursive interaction optimization, especially in large-scale detection tasks. The multi-level detection and recursive refinement process enables the system to quickly and stably perform large-scale target recognition and positioning tasks, avoiding common false positives and false negatives. The recursive refinement mechanism ensures that the detection of each target is based on the optimization results of the previous step, which allows each round of detection to be supplemented and corrected based on the previous results, thereby maintaining the consistency of the results. In addition, the layer-by-layer optimization and path adjustment of the system improves the detection efficiency, allowing it to maintain high performance in large-scale tasks.

[0148] Robustness: The fine-grained detection results generated by the present application cover the precise positioning and spatial relationship of each component of the object, improving the robustness of the detector in different environments and complex scenes. Even in some extremely complex or diverse scenes, the system can still maintain high detection accuracy and stability. By combining the semantic information provided by the visual language model (VLM), the detector can better understand the structure and details of the object, achieving higher adaptability in complex scenes. In addition, the layer-by-layer optimization strategy of chain recursive detection ensures that the target detection process is more robust, effectively avoiding misjudgment and omission in complex environments.

[0149] Enhancing detector performance: The present application trains the detector through high-quality fine-grained detection results, significantly improving the performance of the detector in object recognition and cross-granularity perception tasks. Through continuous optimization and iteration, the performance of the detector is enhanced, especially in fine operation tasks. High-quality fine-grained labeling data is used to train the detector, which can achieve higher precision in object recognition and component positioning. Fine-grained detection results provide more abundant training samples, enhancing the adaptability of the detector to different target levels. By continuously optimizing the detection path of the target and improving the detail recognition ability, the performance of the detector in actual tasks is significantly improved.

[0150] Through experiments, the cross-granularity target detection method and system provided by the present application performs well in cross-granularity scenes and in robot operation tasks, with a success rate of 17.31% higher than traditional detection methods in regular object operations, and a success rate of 51.39% higher in larger object operations, fully demonstrating its effectiveness and reliability in practical applications.

[0151] In summary, the cross-granularity target detection method and system provided by the present application, through the cooperative work of VLM and the detector, combined with the chain recursive detection mechanism, realizes efficient and accurate automatic detection, has wide application prospects and significant technical advantages.

[0152] It should be noted that the cross-granularity target detection system and method provided by the embodiments of the present application can be used in the field of robot technology, and can also be used in any technical field other than the field of robot technology. The embodiments of the present application do not limit the application field of the cross-granularity target detection system and method.

[0153] Based on the same inventive concept, the embodiment of the present application further provides a cross-granularity target detection device which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the cross-granularity target detection device solves the problem by the similar principle as the above method, the implementation of the cross-granularity target detection device can be referred to the implementation of the above method, and the repeated parts will not be described herein. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and is conceived.

[0154] Figure 10 The structural block diagram of the cross-granularity target detection device provided by an embodiment of the present application is shown in Figure 10 The cross-granularity target detection device 04 provided by the embodiment of the present application includes:

[0155] The acquisition module 41 is configured to acquire a target image and target task information.

[0156] The generation module 42 is configured to generate a detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained in order.

[0157] The identification module 43 is configured to identify the position of each detection target in the target image in order by a recursive method according to the order of each detection target in the detection target sequence.

[0158] Figure 11 The structural diagram of the electronic device provided by an embodiment of the present application is shown in Figure 11 The electronic device 05 can include a processor 51, a communications interface 52, a memory 53 and a communications bus 54, wherein the processor 51, the communications interface 52 and the memory 53 complete the communication with each other through the communications bus 54. The processor 51 can invoke the logical instructions in the memory 53 to execute the method described in any of the above embodiments, for example, including: acquiring a target image and target task information; generating a detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained in order; identifying the position of each detection target in the target image in order by a recursive method according to the order of each detection target in the detection target sequence.

[0159] In addition, the logic instructions in the memory 53 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0160] The embodiment application provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the method provided by each method embodiment described above, for example, including: obtaining target image and target task information; generating detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained in order; according to the order of each detection target in the detection target sequence, each detection target is gradually identified in the target image by a recursive method.

[0161] The embodiment provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program enables the computer to execute the method provided by each method embodiment described above, for example, including: obtaining target image and target task information; generating detection target sequence according to the target image and the target task information, wherein the target image contains each detection target, and in the detection target sequence, each detection target is sequentially contained in order; according to the order of each detection target in the detection target sequence, each detection target is gradually identified in the target image by a recursive method.

[0162] Those skilled in the art will appreciate that embodiments of the application can be readily used as a method, a system or a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0163] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0164] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0165] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0166] In the description of the application, reference to terms such as "one embodiment", "one specific embodiment", "some embodiments", "for example", "an example", "a specific example", or "some examples" etc. means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The illustrative appearances of the above-mentioned terms in various places in the specification are not necessarily referred to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0167] The above-described specific embodiments, purposes, technical solutions and beneficial effects of the application are further described in detail. It should be understood that the above-described specific embodiments are only for the purpose of illustrating the application, and are not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A cross-granularity object detection method, characterized in that, The method comprises: obtaining a target image and target task information; generating a detection target sequence according to the target image and the target task information, wherein each detection target is contained in the target image, and each detection target is sequentially contained in the detection target sequence; recursively identifying the position of each detection target in the target image according to the order of each detection target in the detection target sequence; each detection target in the detection target sequence is presented in a textual description; the recursive method for identifying the position of each detection target in the target image according to the order of each detection target in the detection target sequence comprises: inputting the target image and the textual description of the first detection target in the detection target sequence into a target detector to obtain the position information of the first detection target in the target image output by the target detector; cropping the target image according to the position information of the first detection target in the target image to obtain a first sub-image containing the first detection target; inputting the first sub-image and the textual description of the second detection target in the detection target sequence into the target detector to obtain the position information of the second detection target in the first sub-image output by the target detector; cropping the first sub-image according to the position information of the second detection target in the first sub-image to obtain a second sub-image containing the second detection target; and so on, until the position information of the last detection target in the detection target sequence in the Nth sub-image is obtained, where N is a positive integer greater than 2.

2. The method of claim 1, wherein, the method for generating a detection target sequence according to the target image and the target task information comprises: generating a detection target sequence according to the target image and the target task information by a target visual language model, wherein the target visual language model is obtained by training a preset visual language model based on first annotation data, the first annotation data comprises images and task information used as model input, and further comprises a detection target sequence used as a label.

3. The method of claim 2, wherein, the target visual language model generates a detection target sequence according to the target image and the target task information, comprising: performing first visual encoding on the target image to generate a first visual feature vector; performing first language encoding on the target task information to generate a first language feature vector; fusing the first visual feature vector and the first language feature vector by using a first cross-modal attention mechanism to obtain a first joint feature representation; performing language decoding on the first joint feature representation to obtain a detection target sequence.

4. The method of claim 1, wherein, the target detector processes the textual description and image of each input detection target as follows: performing second visual encoding on the image to generate a second visual feature vector; performing second language encoding on the textual description of the detection target to generate a second language feature vector; fusing the second visual feature vector and the second language feature vector by using a second cross-modal attention mechanism to obtain a second joint feature representation; determining a position of the detection target in the image according to the second joint feature representation.

5. The method of claim 1, wherein, Before inputting the target image and a literal description of a first detection target in the detection target sequence into a target detector, the method further comprises: obtaining second annotation data, wherein the second annotation data comprises an image and a literal description of a detection target as model input, and further comprises position information as a label, the position information meeting a target precision requirement; training a preset detector according to the second annotation data to obtain a target detector.

6. A cross-granular target detection apparatus characterized by comprising: comprise: an acquisition module, configured to acquire a target image and target task information; a generation module, configured to generate a detection target sequence according to the target image and the target task information, wherein the target image comprises each detection target, and in the detection target sequence, each detection target is sequentially contained; an identification module, configured to identify a position of each detection target in the target image in a recursive manner according to an order of the detection targets in the detection target sequence; in the detection target sequence, each detection target is presented in a literal description; the identification module is specifically configured to: input the target image and a literal description of a first detection target in the detection target sequence into a target detector to obtain position information of the first detection target in the target image output by the target detector; cut the target image according to the position information of the first detection target in the target image to obtain a first sub-image containing the first detection target; input the first sub-image and a literal description of a second detection target in the detection target sequence into the target detector to obtain position information of the second detection target in the first sub-image output by the target detector; cut the first sub-image according to the position information of the second detection target in the first sub-image to obtain a second sub-image containing the second detection target; and so on, until position information of a last detection target in the detection target sequence in an Nth sub-image is obtained, wherein N is a positive integer greater than 2.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer program product comprises a computer program, which is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images