Training method, detection method and device based on target detection model

Abstract

This application discloses a training method, detection method, and apparatus based on an object detection model. Relating to the fields of artificial intelligence or computer vision, the method includes: acquiring training sample data; training a text encoder and a target visual encoder in the object detection model based on the training sample data to obtain trained text encoders and trained target visual encoders, wherein the target visual encoder includes a hierarchical visual transformer based on a translation window; performing alignment training on a first language encoder and the trained text encoder based on the training sample data to obtain a trained first language encoder; replacing the trained text encoder with the trained first language encoder; and jointly training the trained first language encoder and the trained target visual encoder. This solves the problem in related technologies where object detection models struggle to understand complex semantic instructions, leading to low object detection accuracy.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence or computer vision, and more specifically, to a training method, detection method, and apparatus based on an object detection model. Background Technology

[0002] In related technologies, open-set object detection schemes primarily employ BERT as the language encoder on the language side, relying on token-level features (the smallest semantic unit of text processed by the model) output by BERT for cross-modal alignment. However, since BERT is mainly geared towards general text representation learning, its capabilities in instruction understanding and complex semantic combinations are limited, making it difficult to fully leverage the expressive advantages of long, imperative texts in object description. This results in low object detection accuracy for open-set object detection models when faced with complex instructions.

[0003] Object detection technologies based on closed category sets typically model based on these sets. This means that target categories are predefined during the training phase, and the detection model is trained using supervised learning. These technologies extract image features through convolutional neural networks and use a detection head to regress and classify the target location and category. However, some object detection models based on closed category sets adopt an end-to-end structure, and their detection capability is limited by the category set included in the training phase. For target categories that did not appear in the training or targets with complex semantic combinations, it is difficult to directly and effectively identify them.

[0004] There is currently no effective solution to the problem that target detection models in related technologies have difficulty understanding complex semantic instructions, resulting in low target detection accuracy. Summary of the Invention

[0005] The main purpose of this application is to provide a training method, detection method and device based on an object detection model, so as to solve the problem that the object detection model in the related technology has difficulty understanding complex semantic instructions, resulting in low object detection accuracy.

[0006] To achieve the above objectives, according to one aspect of this application, a training method based on an object detection model is provided. The method includes: acquiring training sample data, wherein the training sample data includes image-text pair data, wherein the image-text pair data includes an image and a text cue that is semantically equivalent to the category label of the image; training a text encoder and a target visual encoder in an object detection model based on the training sample data to obtain a trained text encoder and a trained target visual encoder, wherein the target visual encoder includes a hierarchical visual transformer based on a displacement window, wherein the hierarchical visual transformer based on a displacement window includes a channel attention enhancement mechanism; performing alignment training on a first language encoder and the trained text encoder based on the training sample data to obtain a trained first language encoder, and replacing the trained text encoder with the trained first language encoder in the object detection model, wherein the semantic understanding ability of the first language encoder is higher than that of the text encoder, and the alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder; and jointly training the trained first language encoder and the trained target visual encoder to obtain a trained object detection model.

[0007] Further, the text prompts include: a first text prompt and a second text prompt, wherein the complexity of the first text prompt is lower than that of the second text prompt. Based on the training sample data, the first language encoder and the trained text encoder are aligned and trained to obtain the trained first language encoder, including: based on the first text prompt and a first constraint, the trained text encoder and the first language encoder are aligned and trained to obtain a second language encoder, wherein the first constraint is used to control the text representation output by the first language encoder to be consistent with the text representation output by the trained text encoder during the alignment and training process; based on the second text prompt and the second constraint, the second language encoder is trained to obtain the trained first language encoder, wherein the second constraint is used to improve the semantic understanding ability of the second language encoder.

[0008] Further, the text encoder and target visual encoder in the target detection model are trained based on the training sample data to obtain the trained text encoder and the trained target visual encoder. This includes: training the text encoder and the target visual encoder in a supervised learning manner based on the image in the image text pair data and the first text prompt to obtain the trained text encoder and the trained target visual encoder. The text-image alignment training is used to match the text representation output by the text encoder with the visual representation output by the target visual encoder.

[0009] Furthermore, the hierarchical visual changer based on displacement windows also includes a multilayer perceptron, and the channel attention enhancement mechanism is deployed after the multilayer perceptron.

[0010] Further, acquiring training sample data includes: acquiring image data, wherein the image data includes at least one of the following: labeled images and unlabeled images; if the image data includes the labeled images, extracting classification labels from the labeled images to obtain a first classification label, and generating text prompts semantically equivalent to the first classification label; if the image data includes the unlabeled images, parsing the image content of the unlabeled images using a first language model to obtain a text description, and inputting the text description and preset prompt words into a second language model to obtain a second classification label, and generating text prompts semantically equivalent to the second classification label; combining the text prompts associated with the labeled images and the first classification label, and combining the text prompts associated with the unlabeled images and the second classification label to obtain training sample data.

[0011] Further, the trained first language encoder and the trained target visual encoder are jointly trained to obtain a trained target detection model, including: inputting the second text prompt into the trained first language encoder to extract text features, obtaining first text features; inputting the second text prompt into the trained target visual encoder to extract visual features, obtaining first visual features; performing cross-modal feature fusion on the first text features and the first visual features to obtain a fusion result; using the target detection loss as a supervision signal, backpropagating the fusion result to jointly train the trained first language encoder and the trained target visual encoder to obtain a trained target detection model.

[0012] To achieve the above objectives, according to another aspect of this application, a detection method based on an object detection model is provided. The detection method is applied to a trained object detection model and includes: acquiring a target image captured by an image acquisition terminal; upon receiving a detection task, acquiring second text features, wherein the second text features include one of the following: text features obtained by inputting the detection task into a language encoder in the trained object detection model, and cached text features associated with the detection task; inputting the target image into a visual encoder in the trained object detection model to obtain second visual features; and performing cross-modal feature fusion on the second text features and the second visual features to obtain a recognition result of the target image.

[0013] Furthermore, before acquiring the target image captured by the image acquisition terminal, the method further includes: controlling the image acquisition terminal to perform multi-view image acquisition to obtain multiple images; recognizing the multiple images through the trained target detection model to obtain multi-view recognition results; and setting the image acquisition points of the image acquisition terminal based on the multi-view recognition results.

[0014] To achieve the above objectives, according to another aspect of this application, a training apparatus based on an object detection model is provided. The apparatus includes: a first acquisition unit for acquiring training sample data, wherein the training sample data includes image-text pair data, wherein the image-text pair data includes an image and a semantically equivalent text cue to the category label of the image; a first training unit for training a text encoder and an object visual encoder in the object detection model based on the training sample data, to obtain a trained text encoder and a trained object visual encoder, wherein the object visual encoder includes a hierarchical visual transformer based on a displacement window, the hierarchical visual transformer based on a displacement window including a channel attention enhancement mechanism; and a second training unit. The first language encoder and the trained text encoder are aligned and trained based on the training sample data to obtain a trained first language encoder. In the object detection model, the trained text encoder is replaced by the trained first language encoder, wherein the semantic understanding ability of the first language encoder is higher than that of the text encoder. The alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder. The third training unit is used to jointly train the trained first language encoder and the trained object visual encoder to obtain a trained object detection model.

[0015] Further, the text prompts include: a first text prompt and a second text prompt, wherein the complexity of the first text prompt is lower than that of the second text prompt. The second training unit includes: a first training subunit, used to perform alignment training on the trained text encoder and the first language encoder based on the first text prompt and a first constraint to obtain a second language encoder, wherein the first constraint is used to control the text representation output by the first language encoder to be consistent with the text representation output by the trained text encoder during the alignment training process; and a second training subunit, used to train the second language encoder based on the second text prompt and the second constraint to obtain the trained first language encoder, wherein the second constraint is used to improve the semantic understanding ability of the second language encoder.

[0016] Furthermore, the first training unit includes a third training subunit, used to perform supervised learning training on the text encoder and the target visual encoder based on the image in the image-text pair data and the first text prompt, to obtain the trained text encoder and the trained target visual encoder, wherein the image-text alignment training is used to match the text representation output by the text encoder with the visual representation output by the target visual encoder.

[0017] Furthermore, the hierarchical visual changer based on displacement windows also includes a multilayer perceptron, and the channel attention enhancement mechanism is deployed after the multilayer perceptron.

[0018] Further, the first acquisition unit includes: an acquisition subunit for acquiring image data, wherein the image data includes at least one of the following: labeled images and unlabeled images; a first processing subunit for extracting classification labels from the labeled images to obtain a first classification label, and generating text prompts semantically equivalent to the first classification label, when the image data includes the labeled images; a second processing subunit for parsing the image content of the unlabeled images using a first language model to obtain a text description, and inputting the text description and preset prompt words into a second language model to obtain a second classification label, and generating text prompts semantically equivalent to the second classification label; and a third processing subunit for combining the text prompts associated with the labeled images and the first classification label, and combining the text prompts associated with the unlabeled images and the second classification label to obtain training sample data.

[0019] Further, the third training unit includes: a first input subunit, used to input the second text prompt into the trained first language encoder to extract text features, obtaining first text features; a second input subunit, used to input the second text prompt into the trained target visual encoder to extract visual features, obtaining first visual features; a fusion subunit, used to perform cross-modal feature fusion on the first text features and the first visual features to obtain a fusion result; and a fourth training subunit, used to backpropagate the fusion result using the target detection loss as a supervision signal, to jointly train the trained first language encoder and the trained target visual encoder to obtain a trained target detection model.

[0020] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the training method based on the object detection model.

[0021] According to another aspect of this application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the training method based on the object detection model during runtime.

[0022] According to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the training method based on the object detection model.

[0023] In this application, the following methods are employed: Training sample data is obtained, comprising: image-text pair data, wherein the image-text pair data includes: an image and semantically equivalent text cues to the category labels of that image; based on the training sample data, the text encoder and target visual encoder in the object detection model are trained to obtain the trained text encoder and the trained target visual encoder, wherein the target visual encoder includes: a hierarchical visual transformer based on a displacement window, and the hierarchical visual transformer based on a displacement window includes: a channel attention enhancement mechanism; based on the training sample data, the first language encoder and the trained text encoder are aligned and trained to obtain the trained first language encoder, and in the object detection model, the trained text encoder is replaced by the trained first language encoder, wherein the first language encoder... The semantic understanding capability of the first language encoder is higher than that of the text encoder. Alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder. The trained first language encoder and the trained target visual encoder are jointly trained to obtain the trained target detection model. This solves the technical problem in related technologies where the target detection model is difficult to understand complex semantic instructions, resulting in low target detection accuracy. In this application, the original text encoder is replaced by a language encoder with stronger language understanding capability, and a hierarchical visual transformer based on a displacement window with added channel attention enhancement is used as the visual encoder. This avoids the situation in related technologies where open set target detection models have poor understanding of complex instructions, affecting the target detection accuracy, thereby achieving the technical effect of improving the language understanding capability of the target detection model. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A hardware block diagram of a computer terminal for implementing a training method based on an object detection model is shown.

[0026] Figure 2 This is a flowchart of a training method based on an object detection model provided in an embodiment of this application;

[0027] Figure 3 This is a flowchart illustrating the training of an object detection model according to an embodiment of this application;

[0028] Figure 4 This is a flowchart illustrating the training of the text alignment module according to an embodiment of this application;

[0029] Figure 5This is a flowchart of teacher model training provided according to an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of a target visual encoder provided according to an embodiment of this application;

[0031] Figure 7 This is a flowchart of obtaining training sample data according to an embodiment of this application;

[0032] Figure 8 This is a flowchart of cross-modal joint training provided according to an embodiment of this application;

[0033] Figure 9 This is a flowchart of a target detection model-based detection method provided according to an embodiment of this application;

[0034] Figure 10 This is a flowchart of the model inference module provided according to an embodiment of this application;

[0035] Figure 11 This is a schematic diagram of a training device based on an object detection model provided according to an embodiment of this application;

[0036] Figure 12 This is a schematic diagram of a detection device based on a target detection model provided in an embodiment of this application;

[0037] Figure 13 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws and standards, necessary measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding operation entry points for them to choose to agree to or refuse automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0041] Example 1

[0042] According to an embodiment of this application, a method embodiment for training a target detection model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a training method based on an object detection model is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the target detection model-based training method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned target detection model-based training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0047] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The training method based on the object detection model is shown. Figure 2 This is a flowchart of a training method based on an object detection model according to Embodiment 1 of this application.

[0049] Step S201: Obtain training sample data, wherein the training sample data includes: image-text pair data, wherein the image-text pair data includes: an image and a textual prompt that is semantically equivalent to the category label of the image.

[0050] The training sample data mentioned above may include: data composed of image-text pairs. The image-text pair data may include: images and multiple text expressions that are semantically equivalent or highly related to the target category in the image. For example, the text prompts in the image-text pair may include: a first text prompt and a second text prompt (i.e., a long text prompt). The complexity of the first text prompt (i.e., a short text prompt) may be lower than that of the second text prompt (i.e., a long text prompt). In an optional example, the image-text pair data may also include: bounding boxes, which can be used to indicate the location of the target corresponding to the text prompt in the image-text pair data. For example, if the text prompt is a tree growing by a river, then the bounding box can indicate the location of the tree growing by the river in the image.

[0051] Step S202: Based on the training sample data, train the text encoder and target visual encoder in the target detection model to obtain the trained text encoder and the trained target visual encoder. The target visual encoder includes a hierarchical visual transformer based on a displacement window, and the hierarchical visual transformer based on a displacement window includes a channel attention enhancement mechanism.

[0052] The object detection model mentioned above can be an improved open-set object detection model (GroundingDINO). The improved open-set object detection model can include: an object visual encoder (i.e., an improved visual encoder), a text encoder, a cross-modal fusion module, and a detection head. It can achieve phrase-level object localization through the interaction between text and visual features. The text encoder mentioned above can be the text encoder in the open-set object detection model, such as the pre-trained BERT text encoder.

[0053] The aforementioned target visual encoder can introduce a channel attention enhancement mechanism on the basis of the original Swing Transformer (a hierarchical visual transformer based on a displacement window) to adaptively reweight the feature responses of different channels, so that the network model pays more attention to the features of small target regions with discriminative value.

[0054] In one alternative example, a first text prompt can be used as the language input. The text encoder and object visual encoder in the object detection model are trained through supervised learning to complete the object localization task under textual constraints. This enables the object detection model to establish a stable and consistent cross-modal mapping relationship between object category, spatial location, and visual features. Through the above training process, the text encoder and object visual encoder can output accurate and stable object detection results under short text prompt conditions. Their linguistic features and the visual features extracted by the improved visual encoder form a mature and reusable aligned representation in the cross-modal fusion space.

[0055] Step S203: Based on the training sample data, perform alignment training on the first language encoder and the trained text encoder to obtain the trained first language encoder. In the object detection model, replace the trained text encoder with the trained first language encoder. The semantic understanding ability of the first language encoder is higher than that of the text encoder. The alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder.

[0056] The aforementioned first language encoder can include a language encoder with semantic understanding capabilities superior to that of a text encoder. In an optional example, the first language encoder can be any language model or its lightweight variant that possesses instruction understanding and long text modeling capabilities, including large language models of varying sizes or distilled and quantized language encoders. For example, the first language encoder could be the Qwen3-0.6B language encoder (a lightweight version of a large language model). The aforementioned alignment training can be used to ensure that the text representations (e.g., text features) output by the first language encoder and the trained text encoder are consistent in the representation space.

[0057] In this embodiment, a first language encoder can also be introduced and trained by feature dimension adaptation and language representation alignment, so that the text features output by the first language encoder are consistent with the trained text encoder in the representation space. This allows the text encoder in the object detection model to be stably replaced by a first language encoder with stronger instruction understanding capabilities. By replacing the text encoder, the object detection model can effectively understand the complex semantic information contained in long texts, such as multi-object descriptions, attribute constraints, logical relationships, and exclusion conditions. This enables more accurate open-set object detection without the need for predefined fixed category sets, effectively overcoming the problem of semantic ambiguity and target confusion that easily occur when facing complex natural language descriptions in related technologies, and significantly improving the detection accuracy under complex semantic conditions.

[0058] Step S204: Jointly train the trained first language encoder and the trained target visual encoder to obtain the trained target detection model.

[0059] In this embodiment, the trained first language encoder (i.e., the replaced language encoder in the object detection model) and the trained object visual encoder can be jointly fine-tuned based on text prompts (e.g., second text prompts, i.e., long text prompts) in the training sample data. This enables the object detection model to accurately understand text prompts containing multiple objects, attribute constraints, and complex semantic descriptions. Finally, the trained object detection model can be deployed to the inference module. During the inference phase, only the image and text prompts need to be input to complete object detection, achieving stable localization of open-vocabulary targets under complex semantic conditions.

[0060] Through the above steps, in this embodiment, the original text encoder is replaced with a language encoder that has stronger language understanding capabilities, and a hierarchical visual transformer based on a displacement window with added channel attention enhancement is used as the visual encoder. This avoids the situation in related technologies where open-set object detection models have poor understanding of complex instructions, affecting object detection accuracy. Thus, the technical effect of improving the language understanding capability of the object detection model is achieved. Furthermore, this solves the technical problem in related technologies where object detection models struggle to understand complex semantic instructions, leading to low object detection accuracy.

[0061] Figure 3 This is a flowchart illustrating the training of an object detection model according to an embodiment of this application, such as... Figure 3As shown, the overall process includes stages such as training data construction, teacher model training (i.e., BERT text encoder training and image-text alignment module training), language representation alignment (i.e., long and short text alignment training module), cross-modal joint training (i.e., simultaneously fine-tuning the language encoder and target visual encoder on a long text dataset), and inference application. First, an image-text pair dataset containing short texts and semantically equivalent long texts is constructed through the training data generation module, providing a foundation for the model to learn complex semantics. Then, while maintaining the original detection structure of the open-set object detection model, the BERT text encoder in the original open-set object detection model is trained using short text data, and image-text alignment training is completed to obtain a language-visual feature space fully aligned with the detection task. Based on this, a first language model with stronger complex semantic understanding capabilities is introduced, and through feature dimension adaptation and language representation alignment training, its output text features are consistent with the trained BERT text encoder in the representation space, thus achieving stable replacement of the language encoder. Furthermore, the replaced language encoder and visual encoder are jointly fine-tuned based on a long text dataset, enabling the model to accurately understand text prompts containing multiple objects, attribute constraints, and complex semantic descriptions. Finally, the trained model is deployed to the inference module. During the inference phase, only images and text prompts are needed to complete object detection, achieving stable localization of open-word targets under complex semantic conditions.

[0062] Optionally, in the training method based on the object detection model provided in this application embodiment, the text prompts include: a first text prompt and a second text prompt, wherein the complexity of the first text prompt is lower than that of the second text prompt. Based on training sample data, the first language encoder and the trained text encoder are aligned and trained to obtain the trained first language encoder, including: based on the first text prompt and a first constraint, the trained text encoder and the first language encoder are aligned and trained to obtain a second language encoder, wherein the first constraint is used to control the text representation output by the first language encoder to be consistent with the text representation output by the trained text encoder during the alignment training process; based on the second text prompt and the second constraint, the second language encoder is trained to obtain the trained first language encoder, wherein the second constraint is used to improve the semantic understanding ability of the second language encoder.

[0063] The aforementioned text prompts can include: a first text prompt and a second text prompt. The complexity of the first text prompt can be lower than that of the second text prompt. Optionally, the first text prompt can be called a short text prompt, and the second text prompt can be called a long text prompt. The aforementioned first constraint can be used to control the consistency between the text representation output by the first language encoder and the text representation output by the text encoder during alignment training. Optionally, the first preset constraint can include: a mean squared error loss function (MSE loss). The aforementioned second constraint can be used to improve the semantic understanding ability of the second language encoder. Optionally, the second constraint can include: a margin constraint or a contrast constraint, such as a margin loss.

[0064] To replace the original text encoder in the object detection model with a first-language encoder that has a stronger ability to understand complex instructions, a controlled semantic inheritance and capability extension mechanism is constructed to simultaneously address the issues of model stability and insufficient complex semantic modeling capabilities. In this embodiment, a differentiated training strategy that distinguishes between short and long texts (corresponding to...) is adopted. Figure 3 (The long and short text alignment training module in the middle). Figure 4 This is a flowchart illustrating the training of the text alignment module according to an embodiment of this application, such as... Figure 4 As shown, it includes:

[0065] For short text prompt input, the same first text prompt (i.e., short text prompt) can be simultaneously input into the trained text encoder (e.g., the trained BERT text encoder) and the first language encoder to obtain the corresponding text feature representation. The feature output of both can be constrained by the first constraint (e.g., mean squared error loss function MSE loss) so that the first language encoder can approximate the representation distribution of the trained text encoder as closely as possible in terms of category-level and phrase-level semantic expression. This ensures that the interface consistency and detection performance stability with the cross-modal fusion structure of the original open set object detection model are maintained after the language encoder is replaced.

[0066] For long text prompts, given that the original text encoder in the object detection model (e.g., BERT text encoder) has inherent upper limits in terms of instruction understanding, multi-attribute combination, relation description, and long context modeling, it is no longer necessary to require the first language encoder to maintain the same feature representation as the original text encoder in the object detection model. Instead, by introducing a second constraint (e.g., training based on interval constraints or contrast constraints), the feature distance between the two under long text conditions is intentionally widened to obtain the trained first language encoder. This allows the first language encoder to break free from the limitations of the representation space of the original text encoder in the object detection model and fully learn and express complex semantic information.

[0067] By using a joint training mechanism of "inherited alignment" for short texts and "differential expansion" for long texts, a smooth transition from the original text encoder to a large language model encoder can be achieved. This allows the text features output by the trained first language encoder to not only stably replace the original text encoder in existing detection frameworks, but also significantly improve semantic expression capabilities under long texts and complex instructions. This avoids the problems of representation space mismatch, training instability, and high engineering costs that can be introduced by directly replacing the encoder, thus providing a solid language feature foundation for subsequent cross-modal joint fine-tuning training and open vocabulary target detection driven by complex semantics.

[0068] Optionally, in the training method based on the object detection model provided in this application embodiment, the text encoder and object visual encoder in the object detection model are trained based on training sample data to obtain the trained text encoder and the trained object visual encoder. This includes: performing image-text alignment training on the text encoder and the object visual encoder using supervised learning based on the image and the first text prompt in the image-text pair data to obtain the trained text encoder and the trained object visual encoder. The image-text alignment training is used to match the text representation output by the text encoder with the visual representation output by the object visual encoder.

[0069] The above-mentioned image-text alignment training can complete the task of target localization in images under the constraints of text prompts, enabling the target detection model to form a stable and consistent cross-modal mapping relationship between target category, spatial location and visual features.

[0070] In this embodiment, before introducing a novel language encoder and replacing the structure, a stable and reliable visual-language alignment benchmark model is constructed, thereby providing a reference representation space for subsequent language encoder alignment training and cross-modal joint fine-tuning.

[0071] The teacher model training module can be used to build upon the overall detection framework of the original open-set object detection model. While maintaining its cross-modal fusion mechanism and object detection head structure, the original visual encoder in the original open-set object detection model can be replaced with the improved visual encoder (i.e., the object visual encoder) proposed in this embodiment. The language side still uses the pre-trained BERT text encoder. The improved visual encoder introduces a channel attention enhancement mechanism based on the Swin Transformer structure, enabling it to more effectively extract discriminative visual features related to small targets in mid-to-high-level monitoring scenarios, thus providing the teacher model with a more stable and higher-quality visual representation.

[0072] Figure 5 This is a flowchart of teacher model training provided according to an embodiment of this application, such as... Figure 5As shown, in the specific training process, high-quality image-text pairs from the training sample data can be used, with a focus on category-level or short text prompts as language input. The model is trained through supervised learning to complete the target localization task under textual constraints, enabling the model to establish a stable and consistent cross-modal mapping relationship between target category, spatial location, and visual features. Through the training process, the teacher model can output accurate and stable target detection results under short text prompt conditions. Its language features and the visual features extracted by the improved visual encoder form a mature and reusable alignment representation in the cross-modal fusion space. After training, the network parameters of the teacher model are fixed and used as a reference model in subsequent training stages to guide the feature alignment training of the newly introduced language encoder and as an important benchmark for judging the impact of different text expressions on the detection results. By introducing this teacher model training module, this embodiment effectively avoids the training instability problem caused by the inconsistency between visual and language representations during language encoder replacement, while laying a robust visual-language alignment foundation for subsequent long text semantic modeling and improved target detection capabilities under complex instruction conditions.

[0073] Optionally, in the training method based on the target detection model provided in the embodiments of this application, the hierarchical visual changer based on the displacement window further includes a multilayer perceptron, and the channel attention enhancement mechanism is deployed after the multilayer perceptron.

[0074] In this embodiment, the improved visual encoder in the target visual encoder is used to enhance the feature modeling and representation capabilities of the open vocabulary target detection model (i.e., the open set target detection module) for small-sized targets in mid-to-high-level monitoring scenarios. Figure 6 This is a schematic diagram of a target visual encoder provided according to an embodiment of this application, such as... Figure 6 As shown, Figure 6 As shown, the target visual encoder can adopt a hierarchical Transformer architecture design. The input image is divided into basic feature blocks through Patch Embedding (a technique for image segmentation and conversion into embedding vectors). The original high-dimensional or heterogeneous features are mapped to a low-dimensional semantic space of uniform dimension through linear embedding (such as Linear Embedding). The features are then extracted through multi-stage Patch Merging (for downsampling and building hierarchical features) and Swin Transformer Block (a hierarchical visual transformer module based on displacement windows). The output features of each stage are consistent with the original Swin Transformer structure in the open set target detection model in terms of spatial resolution and channel dimension, thereby ensuring structural compatibility with subsequent cross-modal fusion modules and detection heads.

[0075] like Figure 6As shown, in this embodiment, to address the problem of small targets being easily submerged in complex backgrounds and insufficient channel response, the internal structure of the Swin Transformer Block is improved: In each basic computational unit composed of attention and a Multi-Layer Perceptron (MLP), a channel attention enhancement mechanism based on Squeeze-and-Excitation is introduced after the MLP submodule. This channel attention submodule learns the importance weights of different channels in the current scene through global statistics of features and recalibrates the original features channel by channel, thereby strengthening the discriminative feature channels related to small targets and suppressing redundant responses generated by background regions. This improvement maintains a constant feature tensor size during feature processing. The input features, features after attention calculation, MLP output features, and features enhanced by channel attention all maintain a unified representation of B×H×W×C (i.e., a four-dimensional feature with batch size B, height H, width W, and number of channels C), and the final feature result is output through residual connections without introducing additional spatial downsampling or significant computational overhead. Through the above structural improvements, the visual encoder in this embodiment effectively enhances the model's perception and localization stability of small-scale targets in mid-to-high-level monitoring scenarios while maintaining the original inference efficiency and engineering deployability. This provides higher-quality visual feature input for subsequent cross-modal alignment with the language encoder and target detection tasks.

[0076] In an optional example, the cross-modal fusion method in this embodiment is not limited to the specific implementation in the existing detection framework, and can be replaced by any feature fusion structure based on text-visual interaction; the output task is not limited to object detection, and can also be equivalently extended to instance segmentation, semantic segmentation, object tracking, region retrieval or other visual localization tasks based on text prompts.

[0077] Optionally, in the training method based on the object detection model provided in this application embodiment, obtaining training sample data includes: obtaining image data, wherein the image data includes at least one of the following: labeled images and unlabeled images; when the image data includes labeled images, extracting classification labels from the labeled images to obtain a first classification label, and generating text prompts semantically equivalent to the first classification label; when the image data includes unlabeled images, parsing the image content of the unlabeled images using a first language model to obtain a text description, and inputting the text description and preset prompt words into a second language model to obtain a second classification label, and generating text prompts semantically equivalent to the second classification label; combining the text prompts associated with the labeled images and the first classification label, and combining the text prompts associated with the unlabeled images and the second classification label to obtain training sample data.

[0078] In order to automatically construct high-quality and diverse image-text pair training samples under the condition of limited manual annotation cost, so as to support the language encoder's alignment training for complex semantic prompts and improve the overall performance of the model, in this embodiment, training sample data can be obtained through the training data generation module.

[0079] The image data mentioned above can be images of the Eiffel Tower scene. Figure 7 This is a flowchart of obtaining training sample data according to an embodiment of this application, such as... Figure 7 As shown, it can take images of the Eiffel Tower scene as input, supporting both in-depth mining of labeled image data and automatic utilization of unlabeled image data. The specific implementation process includes:

[0080] For unlabeled images of Eiffel Tower scenes, firstly, a large visual-language model (first language model) with multimodal understanding is used to parse the image content and automatically generate a natural language text description containing the scene, target, and their attribute relationships. Then, through the design of prompt words, the language model (second language model) is guided to parse a set of candidate category labels that can be used for target detection from the generated text description, thereby obtaining multiple potential detectable target categories. The following uses the Eiffel Tower scene as an example to illustrate how to obtain training sample data.

[0081] For labeled images of iron tower scenes, manually labeled target category tags and corresponding ground truth bounding boxes can be directly read as basic supervision signals. Building upon this, a language model can be introduced using the training data generation module to further extend the semantic processing of the tags. For each target category, short text prompts (i.e., first text prompts) and long text prompts (i.e., second text prompts) that are semantically equivalent or highly related can be generated. The long text prompts can include information such as the target's appearance features, spatial location, attribute constraints, or exclusion conditions to simulate complex command inputs in real-world application scenarios. Simultaneously, the generated text prompts and corresponding images can be input into a vision-language model to predict the coordinates of candidate bounding boxes for each target category. Consistency judgment is then performed with existing ground truth bounding boxes to filter out samples with recognition deviations due to differences in textual expression, which are then introduced into the training sample data as difficult samples. Through this method, a diverse set of image-text pairs (i.e., training sample data) is ultimately formed, containing multiple textual expressions (short text prompts and long text prompts) for the same image and their bounding box annotation information. This provides a stable and scalable data source for subsequent language encoder alignment training, cross-modal joint fine-tuning, and improvement of target detection capabilities under complex semantic prompts. This module can automatically expand and control the quality of training samples while significantly reducing the workload of manual annotation, forming an iteratively updated data loop, and providing effective support for the continuous optimization of open vocabulary target detection models.

[0082] In one alternative example, the language model used to generate long text training samples is not limited to a specific model, and the generation strategy is not limited to short text expansion methods. It can be replaced by text generation methods based on templates, knowledge bases, or rules. The sample selection criteria and training process are not limited to a two-stage form. Multi-stage, alternating training, or joint training processes can be adopted according to application requirements.

[0083] Optionally, in the training method based on the object detection model provided in this application embodiment, the trained first language encoder and the trained object visual encoder are jointly trained to obtain the trained object detection model, including: inputting a second text prompt into the trained first language encoder to extract text features to obtain first text features; inputting the second text prompt into the trained object visual encoder to extract visual features to obtain first visual features; performing cross-modal feature fusion on the first text features and the first visual features to obtain a fusion result; using the object detection loss as a supervision signal, backpropagating the fusion result to jointly train the trained first language encoder and the trained object visual encoder to obtain the trained object detection model.

[0084] In this embodiment, a cross-modal joint training module can be set up to jointly fine-tune the language encoder and visual encoder based on long text semantic prompts, thereby giving full play to the advantages of large language models in complex semantic understanding.

[0085] In one alternative example, after completing the language encoder alignment training, the target detection model can be jointly optimized end-to-end with the first language encoder, which has already completed short text alignment and long text differentiation training, and the target visual encoder (i.e., the improved visual encoder) as the core components, while keeping the original cross-modal feature fusion structure and target detection head unchanged. Figure 8 This is a flowchart of cross-modal joint training provided according to an embodiment of this application, such as... Figure 8As shown, in the specific training process, long text prompts containing multiple objects, multiple attribute constraints, spatial relationship descriptions, and exclusion conditions can be used as language input. Images are input into an improved visual encoder to extract multi-scale visual features, while the long text is input into the trained first language encoder to generate high-level semantic features. A cross-modal feature fusion module maps the language features and visual features to a unified semantic space, achieving target candidate region modeling and matching under complex text conditions. By using the target detection loss as a supervision signal, the cross-modal fusion results are backpropagated, enabling the first language encoder and the target visual encoder to collaboratively adjust parameter distributions under the drive of the detection task objective, thereby gradually establishing a stable correspondence between the long text semantics and the visual target. Since this stage no longer relies on the BERT text encoder as a semantic reference, the target detection model can break free from the limitations of its original language representation capabilities and fully learn the semantic differences and discriminative features under complex instruction conditions. Through the cross-modal joint fine-tuning training centered on long texts, the trained object detection model was obtained. While maintaining the stability of short text detection, the model significantly improved the object recall and localization accuracy driven by complex natural language prompts, providing a unified, efficient and robust cross-modal feature representation for open-vocabulary object detection based on long text instructions in the inference stage.

[0086] In this embodiment, the BERT text encoder in the original open-set object detection model is replaced with a language encoder that has stronger language understanding capabilities. Combined with language alignment and distillation training mechanisms, the model can effectively understand complex semantic information such as multi-object descriptions, attribute constraints, logical relationships and exclusion conditions contained in long texts. Thus, it can achieve more accurate open-set object detection without the need for predefined fixed category sets. This effectively overcomes the problem of semantic ambiguity and target confusion that easily occur when facing complex natural language descriptions in related technologies, and significantly improves the detection accuracy under complex semantic conditions.

[0087] To address the issues of small target size, large background proportion, and easily obscured details in high-altitude monitoring scenarios such as iron towers, this embodiment makes targeted improvements to the visual encoder structure. Based on the original Transformer feature modeling, a channel attention enhancement mechanism is introduced to adaptively reweight the feature responses of different channels, making the network pay more attention to small target region features with discriminative value. This significantly reduces the probability of missing detection of distant, small-sized targets and improves the practicality of the model in high-altitude monitoring scenarios.

[0088] By deploying a language alignment module with differentiated constraints for short and long texts, the consistency of the representation spaces of the first language encoder and the BERT text encoder—which have stronger semantic understanding capabilities—is constrained through methods such as mean squared error in short text scenarios to maintain the original cross-modal detection capabilities. In long text scenarios, the consistency constraint is weakened, guiding the model to learn more discriminative semantic representations. This strategy effectively avoids the cross-modal mismatch problem caused by directly replacing the language model, making the cross-modal feature fusion process more stable and reliable.

[0089] In the cross-modal joint training phase, long text descriptions are used as the primary supervision signal to guide the model in learning the deep correlation between "text-visual-spatial location." This enables the model to maintain stable recognition results even when faced with significantly varied task descriptions, complex combinations of target categories, or insufficient prior information. Thus, it effectively addresses the problem of insufficient generalization ability in complex task-driven scenarios in related technologies.

[0090] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0091] Example 2

[0092] According to an embodiment of this application, a method embodiment of a detection method based on an object detection model is also provided. The detection method based on the object detection model is applied to the object detection model trained in Embodiment 1. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0093] This application provides, as follows: Figure 9 The detection method shown is based on the target detection model. Figure 9 This is a flowchart of a target detection model-based detection method according to Embodiment 1 of this application.

[0094] Step S901: Acquire the target image captured by the image acquisition terminal.

[0095] The aforementioned image acquisition terminals may include: tower PTZ cameras (e.g., intelligent monitoring spherical cameras deployed on communication towers), which can be used for business scenarios such as regional patrols, ecological monitoring, emergency response, and water conservancy inspections. Target images may refer to image content containing the target to be detected (such as vehicles, personnel, water bodies, illegal buildings, fire points, etc.).

[0096] Step S902: After receiving the detection task, obtain the second text feature, wherein the second text feature includes one of the following: text features obtained after inputting the detection task into the language encoder in the trained target detection model, and text features associated with the cached detection task.

[0097] The aforementioned detection tasks can be used to indicate the detection tasks to be performed on the target to be detected. Figure 10 This is a flowchart of the model inference module provided according to an embodiment of this application, such as... Figure 10 As shown in this embodiment, after receiving the detection task, the system can parse the specified detection category or target description information in the detection task and determine whether the current task is the first inference. When it is determined to be the first inference, the trained first language encoder and its subsequent fully connected mapping layer (FC) can be invoked to encode the text description corresponding to the detection category, generate a standardized text feature vector, and cache the text feature locally to avoid the computational overhead caused by subsequent repeated encoding. In non-first inference scenarios, the corresponding text feature can be directly read from the cache to achieve a fast response on the text side, thereby obtaining the second text feature.

[0098] Step S903: Input the target image into the visual encoder of the trained target detection model to obtain the second visual feature;

[0099] like Figure 10 As shown, in this embodiment, the acquired target image is input into the trained improved visual encoder (i.e., the trained target visual encoder) to extract high-resolution, multi-scale visual feature representations and obtain the second visual features.

[0100] In one optional example, the model inference module can input cached or real-time generated text features (second text features) and visual features of the current frame (second visual features) into the cross-modal feature fusion unit.

[0101] Step S904: Perform cross-modal feature fusion on the second text features and the second visual features to obtain the recognition result of the target image.

[0102] In this embodiment, the association calculation between the second text feature semantics and the second visual feature is completed through a cross-modal alignment and matching mechanism, and finally the recognition result of the corresponding target and its spatial location are output.

[0103] In this embodiment, while ensuring real-time performance, the computational redundancy caused by repeated text encoding and invalid perspectives is significantly reduced. This enables the system to stably and efficiently complete target recognition tasks in complex scenarios such as high points in the tower, dense small targets, and open categories, demonstrating good engineering practicality and large-scale deployment value.

[0104] Optionally, in the target detection model-based detection method provided in this application embodiment, before acquiring the target image acquired by the image acquisition terminal, the method further includes: controlling the image acquisition terminal to perform multi-view image acquisition to obtain multiple images; recognizing the multiple images through the trained target detection model to obtain multi-view recognition results; and setting the image acquisition points of the image acquisition terminal based on the multi-view recognition results.

[0105] In this embodiment, the image acquisition terminal (e.g., a tower PTZ camera) can continuously acquire multi-view images during rotation. Based on the open-set target detection capability of the target detection model trained in Embodiment 1, large-scale preliminary inference of the multiple acquired images is performed to obtain the target response under different viewpoints. By comprehensively analyzing the multi-view inference results, the inference module automatically determines the optimal observation point and controls the image acquisition terminal to be fixed at that point to improve the stability of subsequent recognition. After the point is fixed, the image acquisition terminal can enter the real-time image acquisition stage.

[0106] This embodiment reduces the computational overhead caused by repeated text calculations and invalid perspectives by pre-encoding and caching text features, combined with multi-view acquisition and optimal point selection mechanisms. This enables the system to have higher inference efficiency while ensuring detection accuracy. It is suitable for practical engineering scenarios such as tower PTZ cameras that require long-term online operation and frequent task switching, and has good deployment and application value.

[0107] Example 3

[0108] This application also provides a training device based on an object detection model. It should be noted that the training device based on an object detection model in this application can be used to execute the training method based on an object detection model provided in this application. The following describes the training device based on an object detection model provided in this application.

[0109] According to embodiments of this application, an apparatus for implementing the above-described training method based on an object detection model is also provided, such as... Figure 11 As shown, the device includes: a first acquisition unit 111, a first training unit 112, a second training unit 113, and a third training unit 114.

[0110] The first acquisition unit 111 is used to acquire training sample data, wherein the training sample data includes: image-text pair data, wherein the image-text pair data includes: an image and a text prompt that is semantically equivalent to the category label of the image;

[0111] The first training unit 112 is used to train the text encoder and target visual encoder in the target detection model based on training sample data, so as to obtain the trained text encoder and the trained target visual encoder. The target visual encoder includes a hierarchical visual transformer based on a displacement window, and the hierarchical visual transformer based on a displacement window includes a channel attention enhancement mechanism.

[0112] The second training unit 113 is used to perform alignment training on the first language encoder and the trained text encoder based on training sample data to obtain the trained first language encoder. In the object detection model, the trained text encoder is replaced by the trained first language encoder. The semantic understanding ability of the first language encoder is higher than that of the text encoder. The alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder.

[0113] The third training unit 114 is used to jointly train the trained first language encoder and the trained target visual encoder to obtain the trained target detection model.

[0114] In the training apparatus based on the object detection model provided in this application embodiment, a first acquisition unit 111 can be used to acquire training sample data, wherein the training sample data includes: image-text pair data, wherein the image-text pair data includes: an image and a text prompt that is semantically equivalent to the category label of the image. Based on the training sample data, a first training unit 112 trains the text encoder and the object visual encoder in the object detection model to obtain the trained text encoder and the trained object visual encoder, wherein the object visual encoder includes: a hierarchical visual transformer based on a displacement window, wherein the hierarchical visual transformer based on a displacement window includes... The method includes a channel attention enhancement mechanism. The second training unit 113 performs alignment training on the first language encoder and the trained text encoder based on training sample data to obtain a trained first language encoder. In the object detection model, the trained text encoder is replaced by the trained first language encoder. The first language encoder has a higher semantic understanding capability than the text encoder. The alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder. The third training unit 114 jointly trains the trained first language encoder and the trained object visual encoder to obtain the trained object detection model. This solves the technical problem in related technologies where object detection models struggle to understand complex semantic instructions, leading to low object detection accuracy. In this embodiment, the original text encoder is replaced by a language encoder with stronger language understanding capabilities, and a hierarchical visual transformer based on a displacement window with added channel attention enhancement is used as the visual encoder. This avoids the situation in related technologies where open-set object detection models have poor understanding of complex instructions, affecting object detection accuracy, thereby achieving the technical effect of improving the language understanding capability of the object detection model.

[0115] Optionally, in the training device based on the object detection model provided in this application embodiment, the text prompts include: a first text prompt and a second text prompt, wherein the complexity of the first text prompt is lower than that of the second text prompt; the second training unit includes: a first training subunit, used to perform alignment training on the trained text encoder and the first language encoder based on the first text prompt and the first constraint to obtain a second language encoder, wherein the first constraint is used to control the text representation output by the first language encoder to be consistent with the text representation output by the trained text encoder during the alignment training process; and a second training subunit, used to train the second language encoder based on the second text prompt and the second constraint to obtain a trained first language encoder, wherein the second constraint is used to improve the semantic understanding ability of the second language encoder.

[0116] Optionally, in the training device based on the target detection model provided in the embodiments of this application, the first training unit includes: a third training subunit, used to perform image-text alignment training on the text encoder and the target visual encoder in a supervised learning manner based on the image and the first text prompt in the image-text pair data, to obtain the trained text encoder and the trained target visual encoder, wherein the image-text alignment training is used to match the text representation output by the text encoder with the visual representation output by the target visual encoder.

[0117] Optionally, in the training device based on the target detection model provided in the embodiments of this application, the hierarchical visual changer based on the displacement window further includes a multilayer perceptron, and the channel attention enhancement mechanism is deployed after the multilayer perceptron.

[0118] Optionally, in the training apparatus based on the object detection model provided in this application embodiment, the first acquisition unit includes: an acquisition subunit, used to acquire image data, wherein the image data includes at least one of the following: labeled images and unlabeled images; a first processing subunit, used to extract classification labels from the labeled images when the image data includes labeled images, obtain a first classification label, and generate text prompts semantically equivalent to the first classification label; a second processing subunit, used to parse the image content of the unlabeled images through a first language model when the image data includes unlabeled images, obtain a text description, and input the text description and preset prompt words into a second language model to obtain a second classification label, and generate text prompts semantically equivalent to the second classification label; and a third processing subunit, used to combine the text prompts associated with the labeled images and the first classification label, and combine the text prompts associated with the unlabeled images and the second classification label to obtain training sample data.

[0119] Optionally, in the training device based on the object detection model provided in this application embodiment, the third training unit includes: a first input subunit, used to input the second text prompt into the trained first language encoder to extract text features, thereby obtaining first text features; a second input subunit, used to input the second text prompt into the trained target visual encoder to extract visual features, thereby obtaining first visual features; a fusion subunit, used to perform cross-modal feature fusion on the first text features and the first visual features, thereby obtaining a fusion result; and a fourth training subunit, used to backpropagate the fusion result with the object detection loss as a supervision signal, so as to jointly train the trained first language encoder and the trained target visual encoder, thereby obtaining a trained object detection model.

[0120] It should be noted that the first acquisition unit 111, the first training unit 112, the second training unit 113, and the third training unit 114 mentioned above correspond to steps S201 to S204 in Embodiment 1. Each unit and the corresponding step implement the same instance and application scenario, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0121] Example 4

[0122] This application also provides a detection device based on a target detection model. It should be noted that the target detection model-based detection device of this application can be used to execute the target detection model-based detection method provided in this application. The target detection model-based detection device provided in this application will be described below.

[0123] According to embodiments of this application, an apparatus for implementing the above-described target detection model-based detection method is also provided, such as... Figure 12 As shown, the device includes: a second acquisition unit 121, a third acquisition unit 122, an input unit 123, and a fusion unit 124.

[0124] The second acquisition unit 121 is used to acquire the target image acquired by the image acquisition terminal;

[0125] The third acquisition unit 122 is used to acquire second text features after receiving a detection task, wherein the second text features include one of the following: text features obtained after inputting the detection task into the language encoder in the trained target detection model, and text features associated with the cached detection task.

[0126] Input unit 123 is used to input the target image into the visual encoder of the trained target detection model to obtain the second visual feature;

[0127] The fusion unit 124 is used to perform cross-modal feature fusion on the second text features and the second visual features to obtain the recognition result of the target image.

[0128] In the target detection model-based detection device provided in this application embodiment, the second acquisition unit 121 acquires the target image captured by the image acquisition terminal. After receiving the detection task, the third acquisition unit 122 acquires second text features, which include one of the following: text features obtained by inputting the detection task into the language encoder of the trained target detection model, or cached text features associated with the detection task. The target image is input into the visual encoder of the trained target detection model through the input unit 123 to obtain second visual features. The fusion unit 124 performs cross-modal feature fusion of the second text features and the second visual features to obtain the recognition result of the target image. Target detection using the trained target detection model achieves the technical effect of improving the target detection accuracy of the target detection model.

[0129] Optionally, in the target detection model-based detection device provided in this application embodiment, before acquiring the target image acquired by the image acquisition terminal, the device further includes: controlling the image acquisition terminal to acquire multiple images from multiple perspectives; recognizing the multiple images through the trained target detection model to obtain multi-view recognition results; and setting the image acquisition points of the image acquisition terminal based on the multi-view recognition results.

[0130] It should be noted that the second acquisition unit 121, the third acquisition unit 122, the input unit 123, and the fusion unit 124 mentioned above correspond to steps S901 to S904 in Embodiment 2. Each unit and its corresponding step implements the same instance and application scenario, but is not limited to the content disclosed in Embodiment 2. It should also be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of a device in a computer terminal.

[0131] Example 5

[0132] Embodiments of this application may provide an electronic device. Figure 13 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 13 As shown, the electronic device may include: one or more ( Figure 13 (Only one is shown) Processor 1302, memory 1304, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0133] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring training sample data, wherein the training sample data includes image-text pair data, wherein the image-text pair data includes an image and a text cue that is semantically equivalent to the category label of the image; training the text encoder and target visual encoder in the object detection model based on the training sample data to obtain the trained text encoder and the trained target visual encoder, wherein the target visual encoder includes a hierarchical visual transformer based on a displacement window, and the hierarchical visual transformer based on a displacement window includes a channel attention enhancement mechanism; performing alignment training on the first language encoder and the trained text encoder based on the training sample data to obtain the trained first language encoder, and replacing the trained text encoder with the trained first language encoder in the object detection model, wherein the semantic understanding ability of the first language encoder is higher than that of the text encoder, and the alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder; and jointly training the trained first language encoder and the trained target visual encoder to obtain the trained object detection model.

[0135] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: The text prompt includes a first text prompt and a second text prompt, the first text prompt being less complex than the second text prompt; based on training sample data, alignment training is performed on the first language encoder and the trained text encoder to obtain the trained first language encoder, including: based on the first text prompt and a first constraint, alignment training is performed on the trained text encoder and the first language encoder to obtain the second language encoder, wherein the first constraint is used to control the text representation output by the first language encoder to be consistent with the text representation output by the trained text encoder during the alignment training process; based on the second text prompt and the second constraint, the second language encoder is trained to obtain the trained first language encoder, wherein the second constraint is used to improve the semantic understanding ability of the second language encoder.

[0136] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: training the text encoder and target visual encoder in the target detection model based on training sample data to obtain the trained text encoder and the trained target visual encoder, including: training the text encoder and the target visual encoder in a supervised learning manner based on the image and the first text prompt in the image text pair data to obtain the trained text encoder and the trained target visual encoder, wherein the image text alignment training is used to match the text representation output by the text encoder with the visual representation output by the target visual encoder.

[0137] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: The hierarchical visual changer based on the displacement window also includes: a multilayer perceptron, with the channel attention enhancement mechanism deployed after the multilayer perceptron.

[0138] The processor can also invoke information and applications stored in the memory via a transmission device to perform the following steps: acquiring training sample data, including: acquiring image data, wherein the image data includes at least one of the following: labeled images and unlabeled images; if the image data includes labeled images, extracting classification labels from the labeled images to obtain a first classification label, and generating text prompts semantically equivalent to the first classification label; if the image data includes unlabeled images, parsing the image content of the unlabeled images using a first language model to obtain a text description, and inputting the text description and preset prompt words into a second language model to obtain a second classification label, and generating text prompts semantically equivalent to the second classification label; combining the text prompts associated with the labeled images and the first classification label, and combining the text prompts associated with the unlabeled images and the second classification label to obtain training sample data.

[0139] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: jointly training the trained first language encoder and the trained target visual encoder to obtain a trained target detection model, including: inputting a second text prompt into the trained first language encoder to extract text features, obtaining first text features; inputting the second text prompt into the trained target visual encoder to extract visual features, obtaining first visual features; performing cross-modal feature fusion on the first text features and the first visual features to obtain a fusion result; using the target detection loss as a supervision signal, backpropagating the fusion result to jointly train the trained first language encoder and the trained target visual encoder to obtain a trained target detection model.

[0140] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: acquiring the target image acquired by the image acquisition terminal; after receiving the detection task, acquiring the second text feature, wherein the second text feature includes one of the following: text features obtained after inputting the detection task into the language encoder in the trained target detection model, and text features associated with the cached detection task; inputting the target image into the visual encoder in the trained target detection model to obtain the second visual feature; performing cross-modal feature fusion on the second text feature and the second visual feature to obtain the recognition result of the target image.

[0141] The processor can also call the information and application program stored in the memory through the transmission device to perform the following steps: before acquiring the target image acquired by the image acquisition terminal, it also includes: controlling the image acquisition terminal to perform multi-view image acquisition to obtain multiple images; recognizing the multiple images through the trained target detection model to obtain multi-view recognition results; and setting the image acquisition points of the image acquisition terminal based on the multi-view recognition results.

[0142] By adopting the embodiments of this application, the original text encoder is replaced by a language encoder with stronger language understanding capabilities, and a hierarchical visual transformer based on a displacement window with added channel attention enhancement is used as the visual encoder. This avoids the situation in related technologies where open set object detection models have poor understanding of complex instructions, which affects the accuracy of object detection. Thus, the technical effect of improving the language understanding capability of the object detection model is achieved.

[0143] Those skilled in the art will understand that Figure 13 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 13This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 13 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 13 The different configurations shown.

[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0145] Example 4

[0146] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the training method based on the object detection model provided in Embodiment 1.

[0147] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0148] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform training method steps based on an object detection model.

[0149] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0155] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A training method based on an object detection model, characterized in that, include: Obtain training sample data, wherein the training sample data includes: image-text pair data, wherein the image-text pair data includes: an image and a textual prompt that is semantically equivalent to the category label of the image; Based on the training sample data, the text encoder and target visual encoder in the target detection model are trained to obtain the trained text encoder and the trained target visual encoder. The target visual encoder includes a hierarchical visual transformer based on a displacement window, and the hierarchical visual transformer based on a displacement window includes a channel attention enhancement mechanism. Based on the training sample data, the first language encoder and the trained text encoder are aligned and trained to obtain the trained first language encoder. In the object detection model, the trained text encoder is replaced by the trained first language encoder. The semantic understanding ability of the first language encoder is higher than that of the text encoder. The alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder. The trained first language encoder and the trained target visual encoder are jointly trained to obtain a trained target detection model.

2. The training method according to claim 1, characterized in that, The text prompts include: a first text prompt and a second text prompt, wherein the complexity of the first text prompt is lower than that of the second text prompt. Based on the training sample data, the first language encoder and the trained text encoder are aligned and trained to obtain the trained first language encoder, which includes: Based on the first text prompt and the first constraint, the text encoder and the first language encoder are aligned and trained to obtain a second language encoder. The first constraint is used to control the text representation output by the first language encoder to be consistent with the text representation output by the trained text encoder during the alignment training process. Based on the second text prompt and the second constraint, the second language encoder is trained to obtain the trained first language encoder, wherein the second constraint is used to improve the semantic understanding ability of the second language encoder.

3. The training method according to claim 2, characterized in that, Based on the training sample data, the text encoder and target visual encoder in the target detection model are trained to obtain the trained text encoder and the trained target visual encoder, including: Based on the image in the image-text pair data and the first text prompt, supervised learning is used to train the text encoder and the target visual encoder to achieve image-text alignment, resulting in a trained text encoder and a trained target visual encoder. The image-text alignment training is used to match the text representation output by the text encoder with the visual representation output by the target visual encoder.

4. The training method according to claim 1, characterized in that, The hierarchical visual changer based on displacement windows also includes a multilayer perceptron, and the channel attention enhancement mechanism is deployed after the multilayer perceptron.

5. The training method according to claim 1, characterized in that, Obtain training sample data, including: Acquire image data, wherein the image data includes at least one of the following: labeled images and unlabeled images; If the labeled image is included in the image data, the classification label is extracted from the labeled image to obtain a first classification label, and a text prompt that is semantically equivalent to the first classification label is generated. When the image data includes the unlabeled image, the image content of the unlabeled image is parsed by the first language model to obtain a text description, and the text description and preset prompt words are input into the second language model to obtain a second category label, and a text prompt that is semantically equivalent to the second category label is generated; The labeled images and the text prompts associated with the first classification label are combined, and the unlabeled images and the text prompts associated with the second classification label are combined to obtain training sample data.

6. The training method according to claim 2, characterized in that, The trained first language encoder and the trained target visual encoder are jointly trained to obtain a trained target detection model, including: The second text prompt is input into the trained first language encoder to extract text features, thus obtaining the first text features; The second text prompt is input into the trained target visual encoder to extract visual features, thus obtaining the first visual feature; Cross-modal feature fusion is performed on the first text feature and the first visual feature to obtain the fusion result; Using the target detection loss as a supervision signal, the fusion result is backpropagated to jointly train the trained first language encoder and the trained target visual encoder to obtain the trained target detection model.

7. A detection method based on a target detection model, characterized in that, The target detection model-based detection method is applied to the trained target detection model in claims 1 to 6, and the detection method includes: Acquire the target image captured by the image acquisition terminal; Upon receiving a detection task, a second text feature is obtained, wherein the second text feature includes one of the following: text features obtained by inputting the detection task into the language encoder of the trained target detection model, or cached text features associated with the detection task. The target image is input into the visual encoder of the trained target detection model to obtain the second visual feature; Cross-modal feature fusion is performed on the second text feature and the second visual feature to obtain the recognition result of the target image.

8. The detection method according to claim 7, characterized in that, Before acquiring the target image captured by the image acquisition terminal, the process also includes: The image acquisition terminal is controlled to acquire images from multiple perspectives, thereby obtaining multiple images; The trained target detection model is used to identify the multiple images to obtain multi-view recognition results; Based on the multi-view recognition results, the image acquisition points of the image acquisition terminal are set.

9. A training device based on an object detection model, characterized in that, include: The first acquisition unit is used to acquire training sample data, wherein the training sample data includes: image-text pair data, wherein the image-text pair data includes: an image and a text prompt that is semantically equivalent to the category label of the image; The first training unit is used to train the text encoder and target visual encoder in the target detection model based on the training sample data, so as to obtain the trained text encoder and the trained target visual encoder. The target visual encoder includes a hierarchical visual transformer based on a displacement window, and the hierarchical visual transformer based on a displacement window includes a channel attention enhancement mechanism. The second training unit is used to perform alignment training on the first language encoder and the trained text encoder based on the training sample data to obtain the trained first language encoder. In the object detection model, the trained text encoder is replaced by the trained first language encoder. The semantic understanding ability of the first language encoder is higher than that of the text encoder. The alignment training is used to align the text representation output by the first language encoder with the text representation of the trained text encoder. The third training unit is used to jointly train the trained first language encoder and the trained target visual encoder to obtain the trained target detection model.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the training method based on the object detection model according to any one of claims 1 to 6, or when executed, they implement the detection method based on the object detection model according to any one of claims 7 to 8.

Images