Target detection model training method and target detection method

Abstract

Embodiments of the present specification provide a target detection model training method and a target detection method, wherein the target detection model training method comprises: acquiring a sample to-be-detected image and corresponding entity category information, inputting the image and the category into a reference feature extraction model to generate sample image feature information, sample entity feature information and sample entity text feature information. Subsequently, the sample image is input into a target detection model to generate predicted image feature information, predicted entity feature information and predicted entity text feature information. The target detection model parameters are adjusted by comparing the above-mentioned various sample features and various predicted features, and a trained target detection model is obtained. Through the collaborative optimization of the reference feature extraction model and the target detection model, the parameter adjustment is performed by using the sample and predicted feature information, the generalization ability of the model to unseen categories and the adaptability of the model to dynamic detection requirements are improved, and the limitation of the fixed category of the traditional target detection method is effectively broken through.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method for training an object detection model. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision, object detection models have been widely used in many fields, significantly improving the efficiency of image understanding and scene analysis. However, traditional detection methods often rely on predefined fixed object categories, which limits the flexibility of the models in real-world scenarios and makes it difficult to meet users' requirements for dynamic and diverse detection.

[0003] Currently, traditional object detection models, trained on labeled datasets with fixed categories, can effectively solve the problem of detecting objects of predefined categories. However, these models cannot adapt to rapidly changing scene requirements, especially exhibiting significant limitations in detecting unseen categories. Therefore, to address these shortcomings, a new object detection model training method is needed to enable the use of the trained model for object detection. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a method for training an object detection model, an object detection method, and an object detection method applied to a cloud device. One or more embodiments of this specification also relate to an object detection model training device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for training an object detection model is provided, comprising:

[0006] Obtain a sample image to be detected that includes at least one sample entity image, and obtain the sample entity category information corresponding to each sample entity image;

[0007] The sample image to be detected, the entity category information of each sample, and the entity image of each sample are input into the reference feature extraction model to obtain the sample image feature information, entity feature information, and entity text feature information generated by the reference feature extraction model. The sample image feature information is the feature information corresponding to the sample image to be detected, the entity feature information is the feature information corresponding to each entity image, and the entity text feature information is the feature information corresponding to each entity category information.

[0008] The sample image to be detected is input into the target detection model to obtain the predicted image feature information, predicted entity feature information and predicted entity text feature information generated by the target detection model.

[0009] The target detection parameters are adjusted based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information to obtain a trained target detection model.

[0010] According to a second aspect of the embodiments of this specification, a target detection method is provided, comprising:

[0011] Acquire the image to be detected;

[0012] The image to be detected is input into the target detection model to obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model. The target detection model is obtained by training the target detection model according to the steps of the above-described target detection model training method.

[0013] Based on the entity image information of each sample and the entity category information corresponding to each entity image, target detection results are generated.

[0014] According to a third aspect of the embodiments of this specification, a target detection method is provided, applied to a cloud device, comprising:

[0015] A target detection request sent by a receiving end-side device, wherein the target detection request includes an image to be detected;

[0016] The image to be detected is input into the target detection model to obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model. The target detection model is obtained by training the target detection model according to the steps of the above-described target detection model training method.

[0017] Based on the entity image information of each sample and the entity category information corresponding to each entity image, a target detection result is generated and the target detection result is sent to the edge device.

[0018] According to a fourth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0019] Memory and processor;

[0020] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0021] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0022] Memory and processor;

[0023] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0024] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0025] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0026] One embodiment of this specification implements a method that, by inputting a sample image to be detected and its corresponding entity category information into a reference feature extraction model, generates a multimodal feature representation including image features, entity features, and text features. These features comprehensively capture the visual information and semantic associations of the target instance, providing rich basic data support for the model. Simultaneously, by inputting the sample image to be detected into the target detection model, predicted image features and entity features are obtained. Through comparison of sample features and predicted features and parameter adjustment, the model's adaptability gaps to diverse scenarios can be effectively identified, thereby precisely optimizing the model's feature expression capabilities in diverse scenarios during parameter adjustment. Thus, through this process of multimodal feature fusion and dynamic comparative optimization, the target detection model can continuously adjust its parameters to adapt to changes in target features in various scenarios, thereby constructing a target detection model that can dynamically adapt to diverse scenarios. This method not only fully utilizes the multimodal information expression capabilities of the reference feature extraction model but also achieves efficient optimization of the target detection model's parameters, thereby improving the ability to recognize unseen categories and meeting the needs of dynamic scenarios and diverse detection methods. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a target detection model training method provided in one embodiment of this specification;

[0028] Figure 2 This is a schematic diagram illustrating a process for obtaining sample entity category information according to an embodiment of this specification;

[0029] Figure 3 This is a schematic diagram illustrating the training process of an object detection model according to one embodiment of this specification;

[0030] Figure 4 This is a flowchart of a target detection method provided in one embodiment of this specification;

[0031] Figure 5 This is a flowchart illustrating a target detection method applied to a cloud device, as provided in one embodiment of this specification.

[0032] Figure 6 This is an architecture diagram of a target detection system provided in one embodiment of this specification;

[0033] Figure 7 This is a flowchart illustrating the processing procedure of a photographic entity detection method according to one embodiment of this specification;

[0034] Figure 8 This is a schematic diagram of the structure of a target detection model training device provided in one embodiment of this specification;

[0035] Figure 9 This is a structural block diagram of an electronic device provided in one embodiment of this specification;

[0036] Figure 10 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0037] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0038] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0041] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0042] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0043] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0044] Contrastive Language–Image Pretraining (CLIP) is a model that combines visual and language understanding. It maps images and text to the same semantic space through contrastive learning, enabling cross-modal matching and understanding. This model can improve its general understanding of images and text through large-scale data pretraining without relying on task-specific annotations. CLIP has wide applications, covering multiple fields such as image classification, text retrieval, and visual question answering.

[0045] This specification provides a method for training an object detection model, an object detection method, and an object detection method applied to a cloud device. One or more embodiments of this specification also relate to an object detection model training apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0046] See Figure 1 , Figure 1 A flowchart of a target detection model training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0047] Step 102: Obtain a sample image to be detected that includes at least one sample entity image, and obtain the sample entity category information corresponding to each sample entity image.

[0048] In practical applications, the sample image to be detected is an input image containing instances of the target to be detected; the sample entity image is a specific instance image cropped or labeled from the sample image to be detected; and the sample entity category information is the target category labeling information corresponding to the sample entity image, including the category text description and the weight of its corresponding category.

[0049] A sample image to be detected can be understood as a complete image that needs to be input into an object detection model or a reference feature extraction model for analysis. It typically contains multiple object instances and background information. For example, a street scene image containing pedestrians, vehicles, and buildings can serve as a sample image to be detected. It provides the model with global scene information, enabling the model to learn the distinguishing characteristics between objects and the background and extract features from different objects.

[0050] Sample entity images can be understood as images of specific target regions segmented from sample images to be detected using annotation or candidate box generation methods. These entity images typically represent local information of a target instance and are core data for fine-grained feature extraction and classification in object detection tasks. For example, a local image of a single car or pedestrian cropped from a street view image can serve as a sample entity image, used to train a model for accurate target feature extraction.

[0051] Sample entity category information can be understood as the annotation information of the target category corresponding to the sample entity image, including the text description of the category and the category weight. The category text description provides semantic information about the target, such as "car" or "pedestrian," while the category weight reflects the confidence level of the category text. For example, in open vocabulary detection tasks, weights may be dynamically generated based on the similarity between the target entity image and the text description, thereby distinguishing between high-confidence and low-confidence category annotations and improving the model's learning efficiency for pseudo-labels or unseen categories.

[0052] This information, through multimodal feature alignment, provides stronger semantic relevance and classification accuracy for object detection models.

[0053] Furthermore, obtain the sample entity category information corresponding to each sample entity image, including:

[0054] Determine the target sample entity image from among the sample entity images, wherein the target sample entity image is any one of the sample entity images;

[0055] Obtain the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image;

[0056] Based on the entity category text sub-information and the entity category weight sub-information, the sample entity category information corresponding to the target sample entity image is obtained, wherein the sample entity category information includes the entity category text sub-information and the entity category weight sub-information.

[0057] In practical applications, entity category text sub-information is the semantic description information of the instance corresponding to the target sample entity image, reflecting its category attributes; entity category weight sub-information is the confidence score of the corresponding category text, used to measure the credibility of the category text description.

[0058] Entity category textual sub-information can be understood as a semantic description associated with the target instance corresponding to the target sample entity image, used to represent the target's category attribute. This textual information is typically obtained by analyzing category labels or generating pseudo-labels, and the core description is extracted using natural language processing tools. For example, in a street view image, the entity category textual sub-information of a target instance might include textual descriptions such as "car" or "truck". By aligning this textual information with the visual features of the target instance, the model can more accurately understand the semantic connotation of the target category, thereby improving the classification accuracy of the object detection task.

[0059] Entity category weight sub-information can be understood as a quantitative score reflecting the credibility of entity category text sub-information, used to assess the degree of association between the category text description and the target instance. This weight value is usually dynamically generated based on the similarity between the target instance and the category text. For example, if the category text sub-information of a target instance is "car," and the visual features of the target highly match the description of "car," then its entity category weight sub-information may be assigned a high confidence value. This weighting mechanism can effectively reduce the negative impact of noise in pseudo-labels on model learning, helping the model to better extract useful category knowledge from real and pseudo-text labels.

[0060] It should be noted that obtaining the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image can be understood as generating category descriptions and confidence scores by analyzing the semantic or visual features of the target sample entity image, thereby providing semantic guidance and classification basis for subsequent object detection models. Specifically, this can be achieved by directly utilizing existing category information in the target sample entity image; when the category information is clear, extracting the text description of that category and setting its weight to 1 ensures high confidence; or, when category information is missing, generating category text sub-information based on the features of the target sample entity image using a generative model, while simultaneously calculating the similarity score with the target as the category weight value; or, matching the target image using an existing category library to find category texts that are close to the target features and dynamically generating weights. For example, a caption generation model can be used to generate a target description, combined with CLIP (Contrastive Language–Image Pretraining) to calculate the similarity between the region and the text, thereby determining the category text and its weight, etc. This specification does not impose any limitations on this approach.

[0061] By combining entity category textual sub-information and weight sub-information, the model achieves efficient alignment of multimodal features and enhanced semantic understanding.

[0062] Further, obtaining the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image includes:

[0063] When the target sample entity image corresponds to reference entity category text sub-information, the reference entity category text sub-information corresponding to the target sample entity image is determined as the entity category text sub-information corresponding to the target sample entity image, and the entity category weight sub-information corresponding to the target sample entity image is determined to be 1;

[0064] If the target sample entity image does not correspond to reference entity category text sub-information, entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image are generated based on the target sample entity image.

[0065] In practical applications, the reference entity category text sub-information refers to the existing category annotations of the sample entity image. The reference entity category text sub-information can be understood as a textual description of the pre-annotated categories, primarily used to provide a semantic expression of the category to which the sample entity image belongs, thus providing an accurate reference for the training and inference of feature extraction and object detection models.

[0066] Obtaining the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image is a crucial step in training and optimizing the object detection model. Its core objective is to generate accurate category semantic descriptions and confidence information for each target instance. This process, by combining existing and inferred category information, achieves multimodal alignment of category text and visual features, thereby improving the model's understanding of target categories and its generalization performance.

[0067] When the target sample entity image has corresponding reference entity category text sub-information, the existing category annotations are used to directly obtain the text description, and the weight is set to 1. This approach ensures that the semantic description of the labeled sample has high reliability, providing stable category supervision for the model. For example, for an image instance labeled "car", its text sub-information is directly "car", with a weight of 1, which is used to guide the model to learn the strong association between category and visual features.

[0068] When a target sample entity image lacks corresponding reference entity category text sub-information, text sub-information and weight sub-information are dynamically generated by analyzing the visual features of the image itself. Specifically, a generative model (such as a caption generation model) can be used to generate candidate category descriptions, and the similarity between the image and the text can be calculated to derive the weight value. For example, an unlabeled target image might be described as "car" or "truck," and then the most suitable category text can be selected and its weight determined based on similarity. This approach not only supplements the information of unlabeled samples but also enhances the model's detection capability for unseen categories.

[0069] This step combines high-confidence supervision of labeled data with dynamic inference from unlabeled data to form a comprehensive acquisition mechanism for category text sub-information and weight sub-information, laying the foundation for subsequent multimodal alignment and model optimization, while effectively improving the accuracy and robustness of the object detection model.

[0070] Furthermore, based on the target sample entity image, entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image are generated, including:

[0071] At least one entity category text is generated based on the target sample entity image, wherein the entity category text is a semantic descriptor of the category to which the target sample entity image belongs;

[0072] Obtain the entity text image similarity corresponding to each entity category text, wherein the entity text image similarity characterizes the degree of matching between the entity category text and the target sample entity image;

[0073] Based on the similarity between entity text images corresponding to each entity category text, entity category text sub-information corresponding to the target sample entity image is determined in each entity category text;

[0074] The similarity between the entity text images corresponding to the entity category text sub-information is determined as the category weight sub-information corresponding to the target sample entity image.

[0075] In practical applications, entity category text is the semantic description information of the category to which the target entity image belongs, used to represent the characteristics of the target category; entity text-image similarity is the degree of matching between entity category text and target sample entity image, used to quantify the strength of the association between the two.

[0076] Entity category text can be understood as a semantic representation of the category of a target instance, typically presented in natural language. This textual description provides a semantic reference for object detection models, guiding them to align visual features with category labels. For example, for a target instance "car," entity category text might include descriptions such as "sedan," "truck," or "electric car." By generating this category text, the model can better establish the association between visual features and categories, thereby improving classification accuracy.

[0077] Entity-text-image similarity can be understood as a quantitative metric used to measure the degree of association between entity category text and target sample entity images. By calculating the similarity between text and image features, the model can assess whether a specific text description is applicable to the target instance. For example, using the CLIP model to calculate the similarity score between the text "car" and a target image, a high similarity score indicates that the text description likely correctly reflects the category of the target instance. This similarity analysis is particularly important in pseudo-label generation, helping to filter high-quality text labels and reduce the interference of noise on model training.

[0078] In one embodiment provided in this specification, the method for obtaining entity category text sub-information and entity category weight sub-information is as follows: Figure 2 , Figure 2This diagram illustrates a process for obtaining sample entity category information according to an embodiment of this specification. The target sample entity image is used as input. An image feature extraction module extracts feature information from the target image, which guides subsequent category text generation and matching. An image description generation model processes the target sample entity image to generate image description text related to the target, reflecting the initial semantic information of the target instance. Subsequently, a word segmentation step decomposes the image description text into specific semantic units, thereby extracting entity category text related to the target category. Next, a text feature extraction module extracts semantic features from the entity category text to align the text with the image in subsequent steps. Simultaneously, a text encoding module converts the extracted text features into a high-dimensional vector representation suitable for computation, further improving the computational efficiency of semantic representation. After the image encoding features and text encoding features are generated, a similarity comparison module matches them, calculating the similarity between the target sample entity image and the generated text to select category text that best reflects the target instance. Finally, category text sub-information is determined based on the similarity matching results, and the similarity value is used as entity category weight sub-information, ensuring that the generated category information is closely related to the visual features of the target instance. This process achieves multimodal alignment of image features and text features through multi-module collaboration, providing accurate and semantically rich category annotation information for object detection tasks.

[0079] By combining similarity values, the model can dynamically adjust the credibility weights of categorized texts, thereby further optimizing object detection performance.

[0080] Step 104: Input the sample image to be detected, the category information of each sample entity, and the image of each sample entity into the reference feature extraction model to obtain the sample image feature information, sample entity feature information, and sample entity text feature information generated by the reference feature extraction model. The sample image feature information is the feature information corresponding to the sample image to be detected, the sample entity feature information is the feature information corresponding to each sample entity image, and the sample entity text feature information is the feature information corresponding to each sample entity category information.

[0081] In practical applications, the reference feature extraction model is a multimodal feature extraction tool that can generate global features and local entity features of sample images; the sample image feature information is the overall representation of the sample image to be detected extracted by the reference feature extraction model; the sample entity feature information is the fine-grained local features of each sample entity image; and the sample entity text feature information is the semantic embedding representation of the sample entity category information, reflecting the text description and semantic similarity of the target category.

[0082] A reference feature extraction model can be understood as a feature extraction tool with multimodal alignment capabilities, capable of extracting corresponding visual and semantic features from input images, category text, and region instances. For example, a CLIP-based reference feature extraction model can simultaneously process street scene images, such as category descriptions for "cars" and "pedestrians," and generate visual and textual features corresponding to these categories. Through this multimodal feature extraction capability, the model establishes a close correlation between image features and category semantics, providing a strong feature foundation for subsequent object detection tasks.

[0083] Sample image feature information can be understood as a global representation generated by the reference feature extraction model for the input sample image to be detected, reflecting the overall semantic and visual characteristics of the sample image. For example, in street view images, sample image feature information can describe the main target types (such as buildings and roads) contained in the image and the distribution characteristics of these targets. This feature information provides the model with global contextual information, helping to improve the model's overall understanding of the target scene.

[0084] Sample entity feature information can be understood as the local fine-grained visual features extracted from the sample entity image by the reference feature extraction model. These features are a precise description of the sample entity image in visual space. For example, the entity features of a car may include its body shape, color, and edge characteristics relative to the background. By extracting sample entity feature information, the model can more accurately capture the detailed information of the target, thereby improving the resolution of target detection.

[0085] The textual feature information of sample entities can be understood as the text embedding representation after the reference feature extraction model semantically encodes the category information of sample entities. This feature contains semantic descriptions of the categories and similarity information, which guide the model to learn the relationship between categories and visual features. For example, for the category "car", its textual feature information may include descriptive words such as "wheels" and "headlights" as well as confidence values ​​related to these categories. This textual feature provides strong semantic support for multimodal tasks and helps improve the model's generalization ability in open-vocabulary detection.

[0086] Furthermore, the sample image to be detected, the category information of each sample entity, and the image of each sample entity are input into the reference feature extraction model to obtain the sample image feature information, sample entity feature information, and sample entity text feature information generated by the reference feature extraction model, including:

[0087] The sample image to be detected and the image information of each sample entity are input into the image feature extraction module in the reference feature extraction model to obtain the sample image feature information and sample entity feature information generated by the image feature extraction module;

[0088] The sample entity category information is input into the text feature extraction module in the reference feature extraction model to obtain the sample entity text feature information generated by the text feature extraction module.

[0089] In practical applications, the image feature extraction module is a component that extracts global and local features from the input image, mainly used to characterize the visual properties of the target instance; the text feature extraction module is a component that extracts semantic features from text information, mainly used to generate semantic representations of categorical text and align them with image features.

[0090] The image feature extraction module can be understood as the core component for extracting visual information from an image. It generates multi-level feature representations by processing the entire image and the target region. These features include information such as the shape, color, texture, and background relationship of the target. For example, in an image containing multiple target instances, the image feature extraction module can extract the edges of vehicles, the outlines of buildings, and the texture information of roads, providing a comprehensive visual feature description for the target detection model. These features are the key basis for the model to determine the location and category of the target, especially in complex scenes, where they can effectively improve the accuracy and robustness of target recognition.

[0091] The text feature extraction module can be understood as a component used to extract semantic information from categorical text, generating high-dimensional semantic feature representations through embedding of categorical text. These features reflect the semantic content of the category and its relationship with other categories. For example, the text feature extraction module can transform the category descriptions of "car" and "truck" into semantic vectors. These vectors not only contain the literal meaning of the category but also reflect its semantic relationship with the higher-level category of "vehicle". In multimodal tasks, the text feature extraction module plays a particularly important role, as its generated semantic features can be compared with image features, helping the model achieve cross-modal feature alignment and target classification.

[0092] By combining image and text feature extraction modules, the model can achieve comprehensive perception from vision to semantics, thereby improving target detection performance in diverse scenarios.

[0093] Furthermore, the sample image to be detected and the image information of each sample entity are input into the image feature extraction module in the reference feature extraction model to obtain the sample image feature information and the sample entity feature information generated by the image feature extraction module, including:

[0094] The sample image to be detected is input into the image feature extraction module to obtain the sample image feature information generated by the image feature extraction module;

[0095] The image information of each sample entity is input into the image feature extraction module to obtain the sample entity feature sub-information corresponding to the image information of each sample entity generated by the image feature extraction module.

[0096] Based on the feature sub-information of each sample entity, sample entity feature information is generated.

[0097] In practical applications, sample entity feature sub-information is fine-grained visual feature information extracted from a single sample entity image, mainly used to characterize the local characteristics of the target instance; sample entity feature information is a high-level feature representation generated based on the comprehensive feature sub-information of multiple sample entities, reflecting the overall feature content of the target instance.

[0098] For example, sample entity feature sub-information can be understood as a local feature representation extracted for a single sample entity image, with the aim of capturing the detailed characteristics of the target instance. These features typically include the target's edge shape, texture pattern, and information distinguishing it from the background. For instance, for an image containing a car, its sample entity feature sub-information might describe the car's outline, the shape of its headlights, and the color of its body.

[0099] It should be noted that generating sample entity feature information based on the feature sub-information of each sample entity can be understood as comprehensively processing the local features of multiple entities to form a unified global feature representation, thereby improving the model's ability to recognize entity characteristics in complex scenes. Specifically, this can be achieved by concatenating the feature sub-information of each sample entity and arranging them into a high-dimensional feature information, thus generating the sample entity feature information. Furthermore, entity feature sub-information can be understood as feature information that includes multiple entity feature sub-information.

[0100] By acquiring the sample entity feature sub-information corresponding to the image information of each sample entity, the target detection model is provided with accurate instance-level input, enabling it to effectively identify targets in complex scenes and improve the resolution of target detection.

[0101] Step 106: Input the sample image to be detected into the target detection model, and obtain the predicted image feature information, predicted entity feature information and predicted entity text feature information generated by the target detection model.

[0102] In practical applications, the object detection model is a model that can extract features and detect object instances in an input image; the predicted image feature information is the global feature representation generated by the object detection model for the sample image to be detected; the predicted entity feature information is the local feature representation generated by the object detection model for the specific target entity image in the sample image to be detected; and the predicted entity text feature information is the semantic feature representation generated by the object detection model for the specific target entity category in the sample image to be detected.

[0103] An object detection model can be understood as a system that performs object instance detection and feature extraction on an input image. By processing target regions in an image, the model can identify the location, shape, and category of the object and generate feature representations. When dealing with complex scenes, this model can combine global background information and local detail information to provide comprehensive support for the detection task. For example, in an image containing multiple objects, an object detection model can simultaneously identify targets of different categories and output their relevant feature information.

[0104] Predicted image feature information can be understood as a global feature representation extracted by the object detection model from the entire image, reflecting the overall content and layout information of the image. These global features contain the distribution relationship between target instances and the background, providing contextual support for subsequent target feature extraction. For example, in a street scene image containing vehicles and pedestrians, predicted image feature information can describe the approximate locations of roads, buildings, and main targets.

[0105] Predicted entity feature information can be understood as the local feature representations generated by the object detection model for each instance in an image, used to describe the specific visual characteristics of the target. These local features typically include the target's shape, texture, and information distinguishing it from the surrounding background. For example, in a street view image, for a specific vehicle instance, the predicted entity feature information might include the vehicle's appearance, color, and edge information, enabling the model to more accurately identify the target.

[0106] Predictive entity textual features can be understood as semantic descriptions generated by object detection models through feature extraction and matching, used to characterize the category of object instances. These textual features establish a correlation between the visual features of the target and the semantics of the category by aligning the target instance with the category description. For example, for an instance of a vehicle, the predicted entity textual features might indicate that its category description is "car" or "truck," thus supporting subsequent object classification and detection.

[0107] Furthermore, the sample image to be detected is input into the target detection model to obtain the predicted image feature information, predicted entity feature information, and predicted entity text feature information generated by the target detection model, including:

[0108] The sample image to be detected is input into the image encoding module in the target detection model to obtain the predicted image feature information generated by the image encoding module;

[0109] The predicted entity feature information is input into the entity detection module in the target detection model to obtain the predicted entity feature information generated by the entity detection module.

[0110] The predicted entity feature information is input into the text mapping module in the target detection model to obtain the predicted entity text feature information generated by the text mapping module.

[0111] In practical applications, the image encoding module is a component that extracts global and local visual features from the input image to generate the basic feature representation of the image; the entity detection module is a component that locates and identifies target instances from image features to generate instance feature information of the target; and the text mapping module is a component that maps target instance features to semantic text representations to generate text features of the target category.

[0112] The image encoding module can be understood as a core component for extracting visual features from an input image. Its purpose is to convert the pixel information of the image into a high-dimensional feature representation. These features can reflect the global structure and local details of the image, including the distribution relationship between the target and the background, the texture and shape of the target, etc. For example, in street view images, the image encoding module can extract visual features of roads, vehicles, pedestrians, and buildings, providing feature input for subsequent object detection tasks.

[0113] The entity detection module can be understood as a component that locates and identifies specific target instances from image features. Its role is to determine the bounding box of the target and extract instance-level feature information based on the features generated by the image encoding module. For example, for an image containing multiple vehicles, the entity detection module can locate the specific position of each vehicle and generate feature representations describing its shape, size, and location. Through the entity detection module, the model can effectively distinguish different target instances in complex scenes, laying the foundation for classification and semantic mapping.

[0114] The text mapping module can be understood as a component that converts the visual features of a target instance into semantic text features, aligning the target instance with category text descriptions. For example, by processing the target features generated by the entity detection module, the text mapping module can generate text features corresponding to category descriptions such as "car" or "truck." This feature conversion enables the model to fuse visual information with textual semantics, improving the generalization ability of the object detection model in scenarios involving unseen categories and open vocabulary detection. The addition of the text mapping module allows the model to achieve multimodal alignment from visual to semantic, thereby meeting diverse detection needs.

[0115] By employing an object detection model comprised of an image encoding module, an entity detection module, and a text mapping module, predictive image feature information, predictive entity feature information, and predictive entity text feature information are extracted respectively. The model is then adjusted based on the corresponding sample feature information, allowing for comprehensive feature calibration from multiple perspectives, including instance level, category level, and image level. This effectively aligns visual features with semantic information. This approach not only helps the model learn richer, finer-grained target features but also fully utilizes the guiding role of semantic text information. This significantly improves the object detection model's ability to detect and classify target instances and generalize to unseen categories in complex scenes, while also laying a solid foundation for the integration of multimodal information.

[0116] Furthermore, the predicted entity feature information is input into the entity detection module in the target detection model to obtain the predicted entity feature information generated by the entity detection module, including:

[0117] The predicted entity feature information is input to the entity detection module to obtain at least one predicted entity feature sub-information generated by the entity detection module, wherein the predicted entity feature sub-information represents the feature information corresponding to any entity in the image to be detected predicted by the entity detection module.

[0118] Based on the feature sub-information of each predicted entity, the feature information of the predicted entity is obtained.

[0119] In practical applications, predicted entity feature sub-information is a fine-grained feature representation of the target instance generated by the entity detection module, mainly used to describe the local characteristics of a single target instance; predicted entity feature information is a comprehensive feature representation after integrating multiple predicted entity feature sub-information, reflecting the overall characteristics of all target instances in the image to be detected.

[0120] Predicted entity feature sub-information can be understood as an independent feature description generated for each target instance in the image to be detected. Its purpose is to capture the visual details and salient features of individual targets. This sub-information covers low-level features such as the target's shape, texture, and color, as well as its distinguishing characteristics from the background. For example, in an image containing various fruits, the predicted entity feature sub-information for an apple might include its circular outline, red texture, and surface gloss. These features provide instance-level input to the object detection model, enabling it to accurately identify and classify each target.

[0121] It should be noted that obtaining predictive entity feature information based on the feature sub-information of each predicted entity can be understood as comprehensively processing the local features of multiple entities to form a unified global feature representation, thereby improving the model's ability to recognize entity characteristics in complex scenes. Specifically, this can be achieved by concatenating the feature sub-information of each predicted entity and arranging them into a high-dimensional feature information, thus generating the predicted entity feature information.

[0122] By integrating the feature sub-information of each predicted entity to obtain the representation of the predicted entity feature information, the model can improve its ability to grasp the global information of the image, and at the same time enhance its understanding of the distribution and association of targets in complex scenes, thereby meeting the needs of higher-level target detection tasks.

[0123] Step 108: Adjust the parameters of the target detection based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information to obtain the trained target detection model.

[0124] The process of adjusting the parameters of the target detection model based on sample image features, sample entity features, sample entity text features, and predicted image features, predicted entity features, and predicted entity text features is a crucial step in model training. This step dynamically optimizes the model's parameters by comparing sample features and predicted features, thereby improving the model's detection accuracy and classification ability for target instances.

[0125] For example, comparing sample image feature information with predicted image feature information ensures that the object detection model can accurately extract global features of the entire image, capture the distribution relationship between the image background and target instances, and thus improve the model's overall understanding of the scene. By optimizing image feature loss, the model can better capture the contrast characteristics between the target and the background, reducing misjudgments of complex backgrounds. Comparing sample entity feature information with predicted entity feature information further calibrates the model's ability to extract fine-grained features of target instances in local regions. This process ensures that the model can not only correctly detect the location of the target, but also identify the target's specific shape, boundary, and detailed features, thereby improving the model's robustness in complex scenes. Aligning sample entity text feature information with predicted entity text feature information strengthens the object detection model's semantic understanding of target categories. By comparing and optimizing textual and visual features, the model can more accurately associate target instances with their corresponding category descriptions, thereby improving its generalization ability to unseen categories. This is particularly important in scenarios such as open vocabulary detection, as it can meet diverse detection needs.

[0126] By jointly optimizing the aforementioned multimodal feature information, the object detection model can achieve comprehensive calibration from global to local and from visual to semantic perspectives. This step is not only a crucial link in the model's transition from its initial state to training completion, but also a core guarantee for the model's high detection accuracy and strong generalization ability.

[0127] Furthermore, the target detection parameters are adjusted based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information to obtain a trained target detection model, including:

[0128] The model loss value is obtained based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information;

[0129] The parameters of the target detection model are adjusted based on the model loss value to obtain the trained target detection model.

[0130] In practical applications, the model loss value is a quantitative indicator that measures the difference between the prediction results of the object detection model and the real labels. It is mainly used to guide the optimization and adjustment of model parameters. The model parameter adjustment is to dynamically optimize the training process of the model based on the feedback of the model loss value, thereby improving the detection performance and generalization ability of the model.

[0131] The model loss value can be understood as a key metric used to evaluate the accuracy of a model's predictions. It quantifies the model's error by comparing the difference between the sample feature information and the predicted feature information. For example, in object detection tasks, the model loss value may include multiple components such as image feature loss, entity feature loss, and text feature loss, each measuring the degree of matching between the model's predictions and the true values ​​at different levels. If a model predicts an offset bounding box position for the "apple" category or fails to accurately classify the object, its loss value will increase significantly, indicating that the model needs further optimization.

[0132] Model parameter tuning can be understood as optimizing the model's training process based on the model's loss value feedback, in order to improve the model's detection accuracy and robustness. For example, when the model's loss value is high, the model's parameter weights can be adjusted through backpropagation, enabling it to more accurately capture the correlation between sample feature information and predicted feature information in subsequent training. By continuously optimizing the parameters, the model can gradually reduce the loss value and improve its ability to detect and classify targets in complex scenes.

[0133] It should be noted that adjusting the parameters of object detection based on the model's loss value to obtain a trained object detection model can be understood as a process of gradually improving the performance and generalization ability of the object detection model by optimizing the model parameters to reduce the loss value. Specifically, this can be achieved through backpropagation, using gradient descent to update the model parameters layer by layer to minimize the loss value; alternatively, a multi-task loss weight adjustment method can be used to dynamically balance different types of losses, such as assigning appropriate weights among image feature loss, entity feature loss, and text feature loss to ensure coordinated optimization of multimodal features; or a learning rate scheduler can be set to gradually reduce the learning rate to refine parameter adjustments and prevent overfitting in later training stages. This manual does not impose any limitations on this approach.

[0134] The combination of model loss value and parameter adjustment is the core link in the optimization of the object detection model from the initial state to the end, providing a reliable guarantee for high-quality object detection results.

[0135] Further, the model loss value is obtained based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information, including:

[0136] The image feature loss value is obtained based on the sample image feature information and the predicted image feature information;

[0137] The entity feature loss value is obtained based on the sample entity feature information and the predicted entity feature information;

[0138] The text feature loss value is obtained based on the sample entity text feature information and the predicted entity text feature information;

[0139] The model loss value is generated based on the image feature loss value, the entity feature loss value, and the text feature loss value.

[0140] In practical applications, image feature loss is an indicator that measures the difference between the feature information of the sample image and the feature information of the predicted image, and is used to optimize the global visual features of the model; entity feature loss is an indicator that measures the difference between the feature information of the sample entity and the feature information of the predicted entity, and is used to improve the model's ability to represent the local features of the target instance; text feature loss is an indicator that measures the difference between the text feature information of the sample entity and the text feature information of the predicted entity, and is used to improve the model's semantic understanding and multimodal alignment performance.

[0141] Image feature loss can be understood as a metric used to evaluate the prediction accuracy of an object detection model at the global image feature level. It measures the model's ability to capture global visual patterns by comparing the feature information of the sample image with the feature information of the predicted image. For example, in an image containing multiple objects, the model optimizes its understanding of the overall scene through image feature loss, enabling it to more accurately extract the distribution information of the background and target instances, thereby improving the overall detection performance.

[0142] Entity feature loss can be understood as an important metric for evaluating a model's ability to capture local features of each target instance. It measures the accuracy of target instance detection and classification by comparing the entity feature information of the sample with that of the predicted entity feature information. For example, for an instance labeled "apple," the entity feature loss can reflect whether the model accurately captures the apple's detailed features such as edge shape, texture, and color. This metric helps the model better distinguish between similar categories (such as "apple" and "orange"), thereby improving the fine-grained classification performance of the detection.

[0143] Text feature loss can be understood as a metric measuring a model's ability to understand the semantic features of target instances. It optimizes the model's multimodal alignment capability by comparing the text feature information of sample entities with the text feature information of predicted entities. For example, in unlabeled samples, text feature loss allows the model to better associate visual features with semantic descriptions (such as "car" or "truck"), improving its generalization ability to unseen categories. By optimizing text feature loss, the model can more accurately utilize semantic information to guide object detection tasks, making it perform better in open-vocabulary scenarios.

[0144] It should be noted that calculating the image feature loss value can be understood as quantifying the model's performance in global feature extraction by comparing the differences between sample image feature information and predicted image feature information, thereby providing a basis for adjusting model parameters. Specifically, the mean squared error method can be used, calculating the average of the squared differences between sample features and predicted features to evaluate the model's accuracy in capturing global image features; alternatively, cosine similarity can be used to measure the angle between sample features and predicted feature vectors, reflecting the directionality and consistency of global feature matching; furthermore, contrastive loss can be combined to widen the distance between image features of different categories while narrowing the distance between features of the same category, thereby optimizing the model's ability to capture category characteristics. This specification does not impose any limitations on this.

[0145] It is important to note that calculating the entity feature loss can be understood as quantifying the model's ability to extract fine-grained features of target instances by evaluating the difference between sample entity feature information and predicted entity feature information, thereby optimizing the model's detection performance. Specifically, this can be achieved by comparing sub-information one by one to calculate the mean squared error between sample and predicted entity feature sub-information, evaluating the model's accuracy in instance-level feature capture; alternatively, cosine similarity can be used to compare the directional consistency between sample and predicted entity feature sub-information to optimize the model's ability to represent instance features; furthermore, contrastive learning can be combined to improve the model's ability to distinguish target instances by narrowing the distance between similar target features and widening the distance between different target features. This specification does not impose any limitations on this approach.

[0146] Furthermore, calculating the text feature loss value can be understood as quantifying the model's ability in semantic feature alignment and multimodal information fusion by evaluating the difference between the sample entity text feature information and the predicted entity text feature information, thereby optimizing the model's text understanding and classification performance. Specifically, this can be achieved by comparing the category probability distributions of predicted and sample text features based on cross-entropy loss to evaluate the model's accuracy in text classification; or by calculating the cosine similarity between text feature vectors to quantify the semantic consistency between sample and predicted text features, thereby optimizing the model's performance in aligning text and visual information; or by combining a weighted loss method based on weights, using entity category text sub-information and category weight sub-information to adjust the weights of the text feature loss, further reducing the interference of low-confidence text on model training. This specification does not impose any limitations on this.

[0147] In one embodiment provided in this specification, the training method of the object detection model refers to... Figure 3 , Figure 3This diagram illustrates a training process for an object detection model according to one embodiment of this specification. The sample image to be detected includes at least one sample entity image and sample entity category information, providing the input data required for the training process. The sample image to be detected generates predicted image feature information through an image encoding module, which is then compared with the sample image feature information extracted by a sample image feature extraction module to calculate an image feature loss value. The sample entity image further generates sample entity feature information through an image feature extraction module, and the sample entity category information generates sample entity text feature information through a text feature extraction module, used to comprehensively capture the visual and semantic features of the target instance. The predicted entity feature information is generated by an entity detection module, and an entity feature loss value is calculated based on the sample entity feature information. The predicted entity text feature information is generated by a text mapping module and used to compare with the sample entity text feature information to calculate a text feature loss value. By integrating the image feature loss value, entity feature loss value, and text feature loss value, the parameters of the object detection model are comprehensively optimized, enabling it to more accurately capture the detailed features and global relationships of instances, thereby improving the overall performance of the object detection model.

[0148] By calculating the loss values ​​in the three directions mentioned above to adjust the model's parameters, the model can be optimized from multiple perspectives, including instance level, category level, and global image level. This ensures that the model can fully learn fine-grained information and global semantic relationships of various features. This multi-level adjustment approach can effectively improve the model's ability to recognize target instances and significantly improve its generalization performance to unseen categories. It also enables more accurate target detection and classification in complex scenes, thereby enhancing the model's overall detection efficiency and robustness.

[0149] Furthermore, obtaining the entity feature loss value based on the sample entity feature information and the predicted entity feature information includes:

[0150] At least one sample entity feature sub-information is obtained based on the sample entity feature information, and at least one predicted entity feature sub-information is obtained based on the predicted entity feature information;

[0151] The entity feature loss value is obtained based on the entity feature sub-information of each sample and the entity feature sub-information of each predicted entity.

[0152] In practical applications, obtaining entity feature loss values ​​based on sample entity feature information and predicted entity feature information is a crucial step in optimizing object detection models. This step allows the model to accurately measure the difference between sample features and predicted features, thus providing direct guidance for adjusting model parameters. Comparing sample entity feature sub-information with predicted entity feature sub-information helps the model identify shortcomings in feature representation at the instance level, especially when capturing fine-grained features such as the texture, shape, or edges of the target.

[0153] Considering that entity feature information can be generated by concatenating various entity feature sub-informations, obtaining at least one sample entity feature sub-information based on sample entity feature information can be understood as decomposing or extracting the sample feature information to obtain an independent feature representation corresponding to each target entity, thereby achieving finer-grained feature analysis and model optimization. Specifically, this can be achieved by directly segmenting the sample entity feature information by row or column to generate the corresponding sample entity feature sub-information for each entity. Similarly, obtaining at least one predicted entity feature sub-information based on predicted entity feature information can also be achieved by directly segmenting the predicted entity feature information by row or column to generate the corresponding predicted entity feature sub-information for each entity.

[0154] It should be noted that obtaining entity feature loss values ​​based on the feature sub-information of each sample entity and each predicted entity can be understood as evaluating the accuracy of the model in instance-level feature extraction and representation by quantifying the differences between local and global features, thus providing a direct basis for parameter optimization. Specifically, this can be achieved by comparing the local feature information of sample entities with the local feature information of predicted entities one by one, calculating the mean squared error, and quantifying the model's ability to capture instance details; alternatively, by comparing the relationship feature information of sample entities with the relationship feature information of predicted entities, using cosine similarity to evaluate the directional consistency of overall feature matching, thereby optimizing the model's global feature alignment ability in complex scenarios; or by combining a multi-level weighted method to weight and synthesize the local feature loss and global feature loss according to scenario requirements, ensuring that the loss value under a specific task can more accurately guide model training. This specification does not impose any limitations on this.

[0155] Because calculating entity feature loss values ​​can significantly improve a model's ability to classify target instances with fine granularity, and by dynamically adjusting model parameters, it can gradually adapt to instance variations in complex scenes. For example, by comparing samples and prediction sub-information one by one, the model can identify the characteristics of different targets during training and reduce confusion between similar targets. This process also provides a more accurate feature base for multi-class detection, thus exhibiting stronger robustness in multi-object detection tasks.

[0156] The calculation of entity feature loss is directly related to model performance optimization. By aligning features at the instance level, the model can not only accurately detect the target location but also make detailed judgments about its category, shape, and other characteristics. This not only improves the model's classification accuracy but also provides important support for the model's generalization ability in diverse scenarios.

[0157] Furthermore, entity feature loss values ​​are obtained based on the entity feature sub-information of each sample and the entity feature sub-information of each predicted entity, including:

[0158] At least one sample entity local feature information and at least one sample entity relationship feature information are obtained based on the feature sub-information of each sample entity. At least one predicted entity local feature information and at least one predicted entity relationship feature information are also obtained based on the feature sub-information of each predicted entity. The sample entity local feature information represents the features of the sample entity image itself, the sample entity relationship feature information represents the relationship between the sample entity image and other sample entity images in the sample image to be detected, the predicted entity local feature information represents the features of the predicted entity image itself, and the predicted entity relationship feature information represents the relationship between the predicted entity image and other predicted entity images in the predicted image to be detected.

[0159] Based on the local feature information of each sample entity and the local feature information of each predicted entity, the entity local feature loss value is generated;

[0160] Based on the entity relationship feature information of each sample and the entity relationship feature information of each predicted entity, an entity relationship feature loss value is generated;

[0161] The entity feature loss value is obtained based on the entity local feature loss value and the entity relation feature loss value.

[0162] In practical applications, sample entity local feature information is a feature representation describing the details of a local region of the target instance, used to capture fine-grained information about the target; predicted entity local feature information is a local region feature generated by the target detection model, used for comparison and optimization with sample features; entity local feature loss is an indicator that quantifies the difference between sample local features and predicted local features, used to guide the parameter adjustment of the model's local features; sample entity relationship feature information is a feature representation reflecting the relationship between the target instance and the overall scene, used to provide global context information; predicted entity relationship feature information is a global feature generated by the target detection model, used to evaluate the model's ability to capture global patterns; entity relationship feature loss is an indicator that measures the degree of matching between sample global features and predicted global features, used to optimize the model's global feature performance.

[0163] For example, sample entity local feature information can be understood as a feature representation generated for a local region of a target instance, primarily capturing the target's detailed characteristics, such as texture, edges, and color. For instance, in an image containing an apple, sample entity local feature information might include the apple's surface texture and gloss; these details help the model distinguish targets of similar categories. Predicted entity local feature information can be understood as the object detection model's prediction of local features of the target instance. These features are compared with sample local features to evaluate the model's performance in capturing fine-grained features. For instance, the apple features predicted by the model might reflect the sharpness of its edges and color distribution; comparing them with sample local features can identify the model's shortcomings. Entity local feature loss can be understood as quantifying the model's local feature extraction capability by calculating the difference between sample local features and predicted local features. For instance, the model might be biased when detecting the edges of small objects (such as keys); the local feature loss value can guide the model to optimize the extraction of such detailed features.

[0164] Sample entity relationship features can be understood as the contextual features of a target instance within the entire scene, including its relative position, scale, and relationship to the background. For example, in a street view image, sample global features might describe the position of vehicles on the road and their distance from pedestrians. Predicted entity relationship features can be understood as the model-generated global features of the target instance, used to simulate the target's contextual behavior within the entire scene. By comparing sample and predicted global features, the model's understanding of the relationship between the target and the background can be optimized. The entity relationship feature loss can be understood as a metric measuring the difference between sample global features and predicted global features. For example, in multi-object detection scenarios, the global feature loss helps the model better adjust the position and background relationship of target instances, thereby improving detection performance in complex scenes.

[0165] In one embodiment provided in this specification, the local feature loss value and the global feature loss value are calculated as shown in Formula 1:

[0166]

[0167]

[0168]

[0169] Where i and j are the feature information numbers to be operated on; n represents the number of sample entity images in the sample image to be detected. The local feature loss value learns the detailed features of the object by comparing the local feature similarity between each sample and the prediction; To improve the similarity of positive samples (i = j), the goal is to increase their similarity; to improve the similarity of negative samples (i ≠ j), the goal is to decrease their similarity. To represent the local feature information qi of the predicted entity and the local feature information of the sample entity The similarity score; q represents the predicted local feature information of the entity. i Local feature information of sample entities The cosine similarity reflects the directional similarity between two feature vectors; τ ckd The temperature parameter is used to adjust the distribution range of similarity; lower temperatures make similarity more sensitive. The sigmoid function (·) normalizes the cosine similarity result, mapping it to the range [0, 1], and represents the similarity score.

[0170] The global feature loss value is used to measure the difference in the distribution of the relationship between the sample entity relationship feature information and the predicted entity relationship feature information; R q [i,:] represents the predicted entity relationship feature information, reflecting the local feature information q of the predicted entity. i The relationship between R and the local feature information of each predicted entity; e [i,:] represents the relational feature information of the sample entities, reflecting the local feature information of the sample entities. The relationship between the local feature information of each sample entity; softmax(·) represents normalizing the relationship matrix into a probability distribution, used to calculate the difference between distributions; KL represents the KL divergence, used to compare the similarity between two distributions and optimize the alignment of global features; τ rkd This represents the temperature parameter, used to adjust the sensitivity of the softmax(·) operation.

[0171] This combination of local and global features optimizes the model's ability to classify target instances in detail and detect them as a whole.

[0172] Furthermore, based on the sample entity text feature information and the predicted entity text feature information, a text feature loss value is obtained, including:

[0173] Based on the sample entity text feature information, obtain the sample entity category text sub-information and entity category weight sub-information corresponding to each sample entity image, and obtain the predicted entity category text sub-information corresponding to each sample entity image based on the predicted entity text feature information. The sample entity category text sub-information is a semantic description of the instance association corresponding to the sample entity image, and the entity category weight sub-information represents the credibility of the sample entity category text sub-information.

[0174] Based on the sample entity category text sub-information, predicted entity category text sub-information, and entity category weight sub-information corresponding to each sample entity image, a text feature loss value is generated.

[0175] In practical applications, generating text feature loss values ​​can be understood as measuring the degree of matching between sample features and model-predicted text features based on the entity category text sub-information and entity category weight sub-information corresponding to each sample entity image, and improving the model's ability to understand text information by optimizing the loss value. Specifically, the correctness of pseudo-text labels can be weighted using region-text similarity. For example, the weight of the text label can be calculated based on the similarity between the region encoding of the CLIP model and the text label, and higher-confidence labels can be given greater weight in the loss calculation. Alternatively, the weights can be dynamically optimized to eliminate the influence of low-confidence labels on model training. For example, a soft threshold strategy can be used to filter the pseudo-label set, thereby reducing the interference of noisy labels. Furthermore, the distribution differences between sample text features and predicted text features can be evaluated by calculating the KL divergence and incorporating it into the loss function to optimize the overall semantic consistency of the model. This specification does not impose any limitations on this.

[0176] For example, the sample entity category text sub-information and entity category weight sub-information are the entity category text sub-information and entity category weight sub-information corresponding to each sample entity image. They belong to the same concept as the entity category text sub-information and entity category weight sub-information described above. For details not described in detail here, please refer to the description of the entity category text sub-information and entity category weight sub-information described above. They will not be repeated here.

[0177] In one embodiment provided in this specification, the text feature loss value is calculated as shown in Formula 2:

[0178]

[0179] Where i and j are the numbers of the data to be operated on; α represents the text feature loss value, used to optimize the model's ability to classify categories; j For entity category weight sub-information; y i，j To determine whether the i-th sample belongs to the j-th category (with a value of 0 or 1) using the textual sub-information of the sample entity category, denoted as , where is the confidence level of the target detection model for generating text sub-information of the i-th predicted entity category for the j-th category; Focal(·) is the focus loss function, mainly used to handle class imbalance problems, optimizing the model by enhancing the weights of samples that are difficult to distinguish; N is the total number of sample entity images, used to normalize the loss value; M+K is the total number of categories of sample entity category text sub-information, including the text sub-information of the entity category corresponding to the sample entity image with an entity category weight sub-information of 1 and the text sub-information of the entity category corresponding to the sample entity image with an entity category weight sub-information of non-1.

[0180] The scheme implemented in this specification generates a multimodal feature representation by inputting the sample image to be detected, sample entity category information, and sample entity image into a reference feature extraction model. This representation includes sample image features, entity features, and text features, enabling comprehensive capture of the visual information and semantic relationships of target instances. Simultaneously, the sample image to be detected is input into the target detection model to generate predicted image features, predicted entity features, and predicted text features. Through comparative analysis between sample features and predicted features, the parameters of the target detection model are progressively optimized. By combining multimodal features, the method effectively integrates multi-level information from local and global sample features. Furthermore, the semantic guidance capability of text features compensates for the deficiencies of visual features, enabling the detection model to more accurately identify targets in complex scenes. Based on this, the model performance is progressively optimized through the refined decomposition of feature loss values ​​(such as image feature loss, entity feature loss, and text feature loss), significantly improving the model's adaptability to diverse target detection needs and its generalization ability to unseen categories.

[0181] Corresponding to the above method embodiments, this specification also provides target detection method embodiments, see [link to embodiments]. Figure 4 , Figure 4 A flowchart of a target detection method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0182] Step 402: Obtain the image to be detected.

[0183] Step 404: Input the image to be detected into the target detection model, and obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model, wherein the target detection model is trained by the steps of the target detection model training method described above.

[0184] Step 406: Generate target detection results based on the entity image information of each sample and the entity category information corresponding to each entity image.

[0185] In practical applications, the image to be detected is the input image for the target detection task, which usually contains multiple target instances and background information; the target detection result is the detection output of the target detection model for each target instance in the image to be detected, including the target's location, category, and related confidence information.

[0186] The image to be detected can be understood as the original image that needs to be input into the object detection model for analysis; it is the fundamental data source for the entire detection task. These images typically contain rich scene information, including target instances and complex background content. For example, a street scene image may contain multiple target instances such as vehicles, pedestrians, and buildings. These images provide the object detection model with input data from a real-world scene, used to extract target features at a hierarchical level, from the overall to the local.

[0187] Object detection results can be understood as the output generated by an object detection model after processing the image to be detected. Specifically, it includes the bounding box, class label, and confidence score for each object instance. For example, for a street scene image containing multiple vehicles, the object detection results might include the coordinates of each vehicle, the classification result (e.g., "car" or "truck"), and the model's confidence score for that result (e.g., 95%). These detection results are typically used for subsequent high-level tasks such as object tracking, scene understanding, or action analysis. By generating high-quality object detection results, various downstream applications can be supported, meeting the needs of complex scenarios.

[0188] The above is an illustrative scheme of an object detection method according to this embodiment. It should be noted that the technical solution of this object detection method and the technical solution of the object detection model training method described above belong to the same concept. For details not described in detail in the technical solution of the object detection method, please refer to the description of the technical solution of the object detection model training method described above.

[0189] By applying the scheme of the embodiments in this specification, and inputting the image to be detected into the target detection model, sample entity image information and its corresponding entity category information can be generated, comprehensively extracting the visual features and category association information of the target instances. Combined with the multi-dimensional information generated by the target detection model, the category and location features of multiple target instances in the image can be efficiently analyzed, adapting to the needs of complex multi-target scenes. Based on this, through the cooperation of sample entity image information and category information, accurate generation of target detection results is achieved, fully utilizing the model optimization capabilities during the training phase, improving the recognition accuracy and flexibility for diverse targets, and enhancing the model's performance in dynamic detection requirements.

[0190] Corresponding to the above method embodiments, this specification also provides embodiments of target detection methods applied to cloud devices, see [link to documentation]. Figure 5 , Figure 5A flowchart of a target detection method for a cloud device according to an embodiment of this specification is shown, which specifically includes the following steps.

[0191] Step 502: Receive a target detection request sent by the receiving end device, wherein the target detection request includes the image to be detected.

[0192] Step 504: Input the image to be detected into the target detection model, and obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model, wherein the target detection model is trained by the steps of the target detection model training method described above.

[0193] Step 506: Generate target detection results based on the entity image information of each sample and the entity category information corresponding to each entity image, and send the target detection results to the end device.

[0194] In practical applications, cloud devices are remote devices that provide powerful computing capabilities and storage resources, and are mainly responsible for processing complex tasks; edge devices are local devices close to the user, and are mainly used for data collection and interaction with the cloud.

[0195] Cloud devices can be understood as high-performance computing and storage devices deployed in remote data centers, used for centralized processing of complex tasks, storage of massive amounts of data, and provision of remote services. For example, in object detection scenarios, cloud devices can receive images to be detected sent by edge devices, and generate object detection results by efficiently inferring from pre-trained object detection models. These devices typically have powerful computing resources and network connectivity, supporting parallel processing of large-scale tasks while reducing the computational burden on edge devices.

[0196] Edge devices can be understood as terminal hardware devices located close to the user, primarily responsible for data collection and communication with the cloud. Examples include smartphones, cameras, and IoT devices. In object detection tasks, edge devices can acquire image data via cameras and send the image to be detected and related requests to cloud devices for processing. After detection is complete, the edge device receives the detection results returned from the cloud and uses them for subsequent display or local decision-making. This edge-cloud collaborative approach improves detection efficiency while meeting users' real-time and diverse needs.

[0197] The above is an illustrative scheme of a target detection method applied to a cloud device according to this embodiment. It should be noted that the technical solution of the target detection method applied to the cloud device belongs to the same concept as the target detection method described above. For details not described in detail in the technical solution of the target detection method applied to the cloud device, please refer to the description of the technical solution of the target detection method described above.

[0198] The scheme described in this specification involves receiving a target detection request from an edge device and inputting the image to be detected into the target detection model. This generates a multi-level feature description, including sample entity image information and entity category information. This feature description combines the visual characteristics and semantic information of the target instance in the image, enabling accurate target analysis. Furthermore, based on the generated sample entity image information and corresponding entity category information, a target detection result is generated and fed back to the edge device, achieving efficient information exchange from the cloud to the edge. During this process, leveraging the powerful computing capabilities of the cloud, the target detection model fully utilizes high-quality feature extraction and semantic association analysis, significantly improving detection accuracy and response efficiency in dynamic scenes and diverse detection needs, providing fast and accurate detection support for the edge device.

[0199] See Figure 6 , Figure 6 This specification illustrates an architecture diagram of a target detection system according to an embodiment of the present specification. The target detection system may include a client 100 and a server 200.

[0200] Client 100 is used to send a target detection request to server 200, wherein the target detection request includes an image to be detected;

[0201] Server 200 is used to input the image to be detected into the target detection model, obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model, wherein the target detection model is trained by the steps of the target detection model training method described above; generate target detection results based on each sample entity image information and the entity category information corresponding to each entity image information; and send the target detection results to client 100.

[0202] Client 100 is also used to receive target detection results sent by server 200.

[0203] The scheme implemented in this specification involves sending the image to be detected to the server for processing. The system can then use the object detection model to generate sample entity image information and its corresponding entity category information. Based on this information, the server can generate object detection results and return them to the client, realizing a complete interactive process from client input to server processing and result return. This architecture effectively utilizes high-performance computing resources and improves the processing efficiency of object detection by centralizing computation and model deployment on the server. Simultaneously, the client only needs to transmit the image to be detected and receive the detection results, requiring no additional computing power, thus reducing the hardware requirements of the terminal device. Furthermore, by uniformly using a trained and optimized object detection model on the server, the system can accommodate diverse detection needs and adapt to complex scenarios, thereby providing end users with a stable and efficient object detection service.

[0204] The target detection system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In the target detection scenario, server 200 is used to provide target detection services between multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0205] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the target detection scenario, users can publish data streams to server 200 through client 100, server 200 can generate target detection results based on the data stream, and push the target detection results to other clients that have established communication.

[0206] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0207] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on electronic devices and depends on the device or certain apps on the device to run. Electronic devices may have displays and support information browsing, such as personal mobile terminals like mobile phones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0208] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0209] It is worth noting that the target detection method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the target detection method provided in the embodiments of this specification. In other embodiments, the target detection method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0210] The following is in conjunction with the appendix Figure 7 Taking the application of the target detection method provided in this specification in photographic entity detection as an example, the target detection method will be further explained. Among other things, Figure 7The present specification illustrates a process flowchart of a photographic entity detection method according to an embodiment, which specifically includes the following steps.

[0211] Step 702: Obtain multiple sample entity images from the sample image, and the entity category information corresponding to each sample entity image.

[0212] It should be noted that, in order to improve the efficiency of model training, it is necessary to increase the number of entity images in a single sample image. Therefore, for the entity images and entity category text sub-information that were originally present in the sample image, the entity category text sub-information corresponding to the entity image can be considered real, and the weight sub-information included in the entity category information corresponding to the entity image can be determined to be 1. For entity images and entity category information pairs that were not originally present in the sample image, it is necessary to automatically generate entity category text sub-information corresponding to the entity image. Then, the confidence level of the entity category text sub-information corresponding to the entity image is the confidence level corresponding to the generated text sub-information of the entity category, and it can be determined that the weight sub-information included in the entity category information corresponding to the entity image is the confidence level corresponding to the generated entity category text sub-information.

[0213] Step 704: Input the sample image into the image encoding module in the CLIP model to obtain the sample image feature information generated by the image encoding module.

[0214] Step 706: Input each sample entity image into the image encoding module in the CLIP model, obtain the sample entity feature sub-information corresponding to each sample entity image generated by the image encoding module, and concatenate the sample entity feature sub-information to generate sample entity feature information.

[0215] Step 708: Input the entity category text sub-information from each entity category information into the text encoding module in the CLIP model, obtain the sample entity text feature sub-information corresponding to each sample entity image generated by the text encoding module, and concatenate the sample entity text feature sub-information to generate sample text feature information.

[0216] Step 710: Input the sample image into the image encoding module in the target detection model and obtain the predicted image feature information generated by the image encoding module.

[0217] Step 712: Input the above predicted image feature information into the entity detection module in the target detection model to obtain predicted entity feature information including multiple predicted entity feature sub-information.

[0218] Step 714: Input the above predicted entity feature information into the text mapping module in the target detection model to obtain predicted entity text feature information including multiple predicted entity text feature sub-information.

[0219] Step 716: Calculate the image feature loss value based on the above sample image feature information and predicted image feature information, and calculate the entity feature loss value based on the above sample entity feature sub-information and predicted entity feature sub-information.

[0220] Step 718: Calculate the text feature loss value based on the above entity text feature sub-information and entity category text sub-information, and the predicted entity text feature sub-information and entity category text sub-information.

[0221] Step 720: Adjust the parameters of the above object detection model based on the image feature loss value, entity feature loss value and text feature loss value until the model training stops.

[0222] Step 722: Obtain the photo that needs to be detected, input the photo into the target detection model that has been trained above, and obtain the entity location information and category information corresponding to multiple entities in the image generated by the target detection model.

[0223] The scheme implemented in this specification acquires multiple sample entity images and their corresponding entity category information from a sample image, and uses this information to generate entity category text sub-information and weight sub-information, thereby dynamically expanding the coverage of entity categories. Combining the image encoding module and text encoding module of the CLIP model, the sample image, sample entity image, and corresponding entity category text sub-information are encoded into multimodal features, including sample image features, sample entity features, and sample text features. Further, the sample image and the generated features are input into the target detection model to generate predicted feature information, and the target detection model's parameters are finely adjusted through the calculation of image feature loss, entity feature loss, and text feature loss. This process fully combines the complementary advantages of image features and text features, enhancing the model's generalization ability when detecting different categories of entities. Simultaneously, by dynamically generating category information, it improves the model's adaptability in handling complex and dynamic scenes, thereby achieving efficient recognition and localization of multiple entities in an image.

[0224] Corresponding to the above method embodiments, this specification also provides embodiments of a target detection model training device. Figure 8 A schematic diagram of a target detection model training device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0225] The sample acquisition module 802 is configured to acquire a sample image to be detected, which includes at least one sample entity image, and to acquire sample entity category information corresponding to each sample entity image;

[0226] The sample feature extraction module 804 is configured to input the sample image to be detected, the sample entity category information, and the sample entity image into a reference feature extraction model, and obtain the sample image feature information, sample entity feature information, and sample entity text feature information generated by the reference feature extraction model, wherein the sample image feature information is the feature information corresponding to the sample image to be detected, the sample entity feature information is the feature information corresponding to the sample entity image, and the sample entity text feature information is the feature information corresponding to the sample entity category information;

[0227] The feature prediction module 806 is configured to input the sample image to be detected into the target detection model and obtain the predicted image feature information, predicted entity feature information and predicted entity text feature information generated by the target detection model.

[0228] The parameter adjustment module 808 is configured to adjust the parameters of the target detection based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information, so as to obtain a trained target detection model.

[0229] Optionally, the sample acquisition module 802 is further configured to:

[0230] Determine the target sample entity image from among the sample entity images, wherein the target sample entity image is any one of the sample entity images;

[0231] Obtain the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image;

[0232] Based on the entity category text sub-information and the entity category weight sub-information, the sample entity category information corresponding to the target sample entity image is obtained, wherein the sample entity category information includes the entity category text sub-information and the entity category weight sub-information.

[0233] Optionally, the sample acquisition module 802 is further configured to:

[0234] When the target sample entity image corresponds to reference entity category text sub-information, the reference entity category text sub-information corresponding to the target sample entity image is determined as the entity category text sub-information corresponding to the target sample entity image, and the entity category weight sub-information corresponding to the target sample entity image is determined to be 1;

[0235] If the target sample entity image does not correspond to reference entity category text sub-information, entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image are generated based on the target sample entity image.

[0236] Optionally, the sample acquisition module 802 is further configured to:

[0237] At least one entity category text is generated based on the target sample entity image, wherein the entity category text is a semantic descriptor of the category to which the target sample entity image belongs;

[0238] Obtain the entity text image similarity corresponding to each entity category text, wherein the entity text image similarity characterizes the degree of matching between the entity category text and the target sample entity image;

[0239] Based on the similarity between entity text images corresponding to each entity category text, entity category text sub-information corresponding to the target sample entity image is determined in each entity category text;

[0240] The similarity between the entity text images corresponding to the entity category text sub-information is determined as the category weight sub-information corresponding to the target sample entity image.

[0241] Optionally, the sample feature extraction module 804 is further configured to:

[0242] The sample image to be detected and the image information of each sample entity are input into the image feature extraction module in the reference feature extraction model to obtain the sample image feature information and sample entity feature information generated by the image feature extraction module;

[0243] The sample entity category information is input into the text feature extraction module in the reference feature extraction model to obtain the sample entity text feature information generated by the text feature extraction module.

[0244] Optionally, the sample feature extraction module 804 is further configured to:

[0245] The sample image to be detected is input into the image feature extraction module to obtain the sample image feature information generated by the image feature extraction module;

[0246] The image information of each sample entity is input into the image feature extraction module to obtain the sample entity feature sub-information corresponding to the image information of each sample entity generated by the image feature extraction module.

[0247] Based on the feature sub-information of each sample entity, sample entity feature information is generated.

[0248] Optionally, the feature prediction module 806 is further configured to:

[0249] The sample image to be detected is input into the image encoding module in the target detection model to obtain the predicted image feature information generated by the image encoding module;

[0250] The predicted entity feature information is input into the entity detection module in the target detection model to obtain the predicted entity feature information generated by the entity detection module.

[0251] The predicted entity feature information is input into the text mapping module in the target detection model to obtain the predicted entity text feature information generated by the text mapping module.

[0252] Optionally, the feature prediction module 806 is further configured to:

[0253] The predicted entity feature information is input to the entity detection module to obtain at least one predicted entity feature sub-information generated by the entity detection module, wherein the predicted entity feature sub-information represents the feature information corresponding to any entity in the image to be detected predicted by the entity detection module.

[0254] Based on the feature sub-information of each predicted entity, the feature information of the predicted entity is obtained.

[0255] Optionally, the parameter adjustment module 808 is further configured to:

[0256] The model loss value is obtained based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information;

[0257] The parameters of the target detection model are adjusted based on the model loss value to obtain the trained target detection model.

[0258] Optionally, the parameter adjustment module 808 is further configured to:

[0259] The image feature loss value is obtained based on the sample image feature information and the predicted image feature information;

[0260] The entity feature loss value is obtained based on the sample entity feature information and the predicted entity feature information;

[0261] The text feature loss value is obtained based on the sample entity text feature information and the predicted entity text feature information;

[0262] The model loss value is generated based on the image feature loss value, the entity feature loss value, and the text feature loss value.

[0263] Optionally, the parameter adjustment module 808 is further configured to:

[0264] Based on the sample entity text feature information, obtain the sample entity category text sub-information and entity category weight sub-information corresponding to each sample entity image, and obtain the predicted entity category text sub-information corresponding to each sample entity image based on the predicted entity text feature information. The sample entity category text sub-information is a semantic description of the instance association corresponding to the sample entity image, and the entity category weight sub-information represents the credibility of the sample entity category text sub-information.

[0265] Based on the sample entity category text sub-information, predicted entity category text sub-information, and entity category weight sub-information corresponding to each sample entity image, a text feature loss value is generated.

[0266] Optionally, the parameter adjustment module 808 is further configured to:

[0267] At least one sample entity local feature information and at least one sample entity relationship feature information are obtained based on the feature sub-information of each sample entity. At least one predicted entity local feature information and at least one predicted entity relationship feature information are also obtained based on the feature sub-information of each predicted entity. The sample entity local feature information represents the features of the sample entity image itself, the sample entity relationship feature information represents the relationship between the sample entity image and other sample entity images in the sample image to be detected, the predicted entity local feature information represents the features of the predicted entity image itself, and the predicted entity relationship feature information represents the relationship between the predicted entity image and other predicted entity images in the predicted image to be detected.

[0268] Based on the local feature information of each sample entity and the local feature information of each predicted entity, the entity local feature loss value is generated;

[0269] Based on the entity relationship feature information of each sample and the entity relationship feature information of each predicted entity, an entity relationship feature loss value is generated;

[0270] The entity feature loss value is obtained based on the entity local feature loss value and the entity relation feature loss value.

[0271] Optionally, the parameter adjustment module 808 is further configured to:

[0272] Based on the sample entity text feature information, obtain the entity category text sub-information and entity category weight sub-information corresponding to each sample entity image;

[0273] Based on the entity category text sub-information and entity category weight sub-information corresponding to each sample entity image, a text feature loss value is generated.

[0274] The above is a schematic scheme of an object detection model training device according to this embodiment. It should be noted that the technical solution of this object detection model training device and the technical solution of the object detection model training method described above belong to the same concept. For details not described in detail in the technical solution of the object detection model training device, please refer to the description of the technical solution of the object detection model training method described above.

[0275] The scheme implemented in this specification involves a sample acquisition module that collects sample images to be detected and their entity category information, and further generates entity category text and weight information, thereby refining the semantic expression of sample features. A reference feature extraction module generates sample image features, entity features, and text features through multimodal feature extraction, providing support for deep semantic association of features. A feature prediction module, combined with the predicted feature information generated by the object detection model, establishes a multidimensional comparison mechanism between sample features and predicted features. A parameter adjustment module utilizes a loss value decomposition mechanism to optimize the model at the levels of image features, entity features, and text features, effectively improving the model's generalization ability and multi-scene adaptability, while enhancing semantic understanding and detection accuracy for complex targets. The overall process achieves efficient integration of multimodal features and dynamic adjustment of the object detection model, significantly improving detection capabilities.

[0276] Figure 9 A structural block diagram of an electronic device 900 according to an embodiment of this application is shown. The components of the electronic device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0277] Electronic device 900 also includes access device 940, which enables electronic device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 900 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0278] In one embodiment of this application, the above-mentioned components of the electronic device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The illustrated electronic device block diagram is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0279] Electronic device 900 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 900 can also be a mobile or stationary server.

[0280] The processor 920 is used to execute the following computer program / instruction, which, when executed by the processor, implements the above-mentioned target detection model training method and the steps of the target detection method.

[0281] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solutions of the target detection model training method and the target detection method described above. For details not described in detail in the technical solution of the electronic device, please refer to the description of the technical solutions of the target detection model training method and the target detection method described above.

[0282] Figure 10 A structural block diagram of a computing device 1000 provided according to one embodiment of this specification is shown.

[0283] The computing device 1000 includes:

[0284] Memory 1010 and processor 1020;

[0285] The memory 1010 is used to store computer programs / instructions, and the processor 1020 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1020, they implement the steps of the target detection model training method, the target detection method, and the target detection method applied to cloud devices.

[0286] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0287] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0288] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0289] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the above-described object detection model training method and object detection method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the above-described object detection model training method and object detection method.

[0290] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described target detection model training method and target detection method.

[0291] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiment is described simply because it is substantially similar to the object detection model training method and the object detection method embodiment; relevant parts can be referred to the descriptions of the object detection model training method and the object detection method embodiment.

[0292] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described target detection model training method and target detection method.

[0293] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-described target detection model training method and target detection method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-described target detection model training method and target detection method.

[0294] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0295] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0296] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0297] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0298] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training an object detection model, comprising: Obtain a sample image to be detected that includes at least one sample entity image, and obtain the sample entity category information corresponding to each sample entity image; The sample image to be detected, the entity category information of each sample, and the entity image of each sample are input into the reference feature extraction model to obtain the sample image feature information, entity feature information, and entity text feature information generated by the reference feature extraction model. The sample image feature information is the feature information corresponding to the sample image to be detected, the entity feature information is the feature information corresponding to each entity image, and the entity text feature information is the feature information corresponding to each entity category information. The sample image to be detected is input into the target detection model to obtain the predicted image feature information, predicted entity feature information and predicted entity text feature information generated by the target detection model. The target detection parameters are adjusted based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information to obtain a trained target detection model.

2. The method as described in claim 1, wherein obtaining sample entity category information corresponding to each sample entity image includes: Determine the target sample entity image from among the sample entity images, wherein the target sample entity image is any one of the sample entity images; Obtain the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image; Based on the entity category text sub-information and the entity category weight sub-information, the sample entity category information corresponding to the target sample entity image is obtained, wherein the sample entity category information includes the entity category text sub-information and the entity category weight sub-information.

3. The method as described in claim 2, wherein obtaining the entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image includes: When the target sample entity image corresponds to reference entity category text sub-information, the reference entity category text sub-information corresponding to the target sample entity image is determined as the entity category text sub-information corresponding to the target sample entity image, and the entity category weight sub-information corresponding to the target sample entity image is determined to be 1; If the target sample entity image does not correspond to reference entity category text sub-information, entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image are generated based on the target sample entity image.

4. The method as described in claim 3, wherein generating entity category text sub-information and entity category weight sub-information corresponding to the target sample entity image based on the target sample entity image, includes: At least one entity category text is generated based on the target sample entity image, wherein the entity category text is a semantic descriptor of the category to which the target sample entity image belongs; Obtain the entity text image similarity corresponding to each entity category text, wherein the entity text image similarity characterizes the degree of matching between the entity category text and the target sample entity image; Based on the similarity between entity text images corresponding to each entity category text, entity category text sub-information corresponding to the target sample entity image is determined in each entity category text; The similarity between the entity text images corresponding to the entity category text sub-information is determined as the category weight sub-information corresponding to the target sample entity image.

5. The method according to any one of claims 1-4, wherein the sample image to be detected, the category information of each sample entity, and the image of each sample entity are input into a reference feature extraction model to obtain sample image feature information, sample entity feature information, and sample entity text feature information generated by the reference feature extraction model, comprising: The sample image to be detected and the image information of each sample entity are input into the image feature extraction module in the reference feature extraction model to obtain the sample image feature information and sample entity feature information generated by the image feature extraction module; The sample entity category information is input into the text feature extraction module in the reference feature extraction model to obtain the sample entity text feature information generated by the text feature extraction module.

6. The method as described in claim 5, wherein the sample image to be detected and the image information of each sample entity are input into the image feature extraction module in the reference feature extraction model to obtain the sample image feature information and the sample entity feature information generated by the image feature extraction module, comprising: The sample image to be detected is input into the image feature extraction module to obtain the sample image feature information generated by the image feature extraction module; The image information of each sample entity is input into the image feature extraction module to obtain the sample entity feature sub-information corresponding to the image information of each sample entity generated by the image feature extraction module. Based on the feature sub-information of each sample entity, sample entity feature information is generated.

7. The method according to any one of claims 1-4, wherein the sample image to be detected is input into a target detection model, and the predicted image feature information, predicted entity feature information, and predicted entity text feature information generated by the target detection model are obtained, comprising: The sample image to be detected is input into the image encoding module in the target detection model to obtain the predicted image feature information generated by the image encoding module; The predicted entity feature information is input into the entity detection module in the target detection model to obtain the predicted entity feature information generated by the entity detection module. The predicted entity feature information is input into the text mapping module in the target detection model to obtain the predicted entity text feature information generated by the text mapping module.

8. The method of claim 7, wherein the predicted entity feature information is input into the entity detection module in the target detection model to obtain the predicted entity feature information generated by the entity detection module, comprising: The predicted entity feature information is input to the entity detection module to obtain at least one predicted entity feature sub-information generated by the entity detection module, wherein the predicted entity feature sub-information represents the feature information corresponding to any entity in the image to be detected predicted by the entity detection module. Based on the feature sub-information of each predicted entity, the feature information of the predicted entity is obtained.

9. The method as described in claims 1-4, wherein the target detection parameters are adjusted based on the sample image feature information, the sample entity feature information, and the sample entity text feature information, as well as the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information, to obtain a trained target detection model, comprising: The model loss value is obtained based on the sample image feature information, the sample entity feature information, the sample entity text feature information, the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information; The parameters of the target detection model are adjusted based on the model loss value to obtain the trained target detection model.

10. The method of claim 9, wherein obtaining the model loss value based on the sample image feature information, the sample entity feature information, and the sample entity text feature information, as well as the predicted image feature information, the predicted entity feature information, and the predicted entity text feature information, comprises: The image feature loss value is obtained based on the sample image feature information and the predicted image feature information; The entity feature loss value is obtained based on the sample entity feature information and the predicted entity feature information; The text feature loss value is obtained based on the sample entity text feature information and the predicted entity text feature information; The model loss value is generated based on the image feature loss value, the entity feature loss value, and the text feature loss value.

11. The method of claim 10, wherein obtaining the entity feature loss value based on the sample entity feature information and the predicted entity feature information comprises: At least one sample entity feature sub-information is obtained based on the sample entity feature information, and at least one predicted entity feature sub-information is obtained based on the predicted entity feature information; The entity feature loss value is obtained based on the entity feature sub-information of each sample and the entity feature sub-information of each predicted entity.

12. The method of claim 11, wherein obtaining entity feature loss values ​​based on the feature sub-information of each sample entity and the feature sub-information of each predicted entity includes: At least one sample entity local feature information and at least one sample entity relationship feature information are obtained based on the feature sub-information of each sample entity. At least one predicted entity local feature information and at least one predicted entity relationship feature information are also obtained based on the feature sub-information of each predicted entity. The sample entity local feature information represents the features of the sample entity image itself, the sample entity relationship feature information represents the relationship between the sample entity image and other sample entity images in the sample image to be detected, the predicted entity local feature information represents the features of the predicted entity image itself, and the predicted entity relationship feature information represents the relationship between the predicted entity image and other predicted entity images in the predicted image to be detected. Based on the local feature information of each sample entity and the local feature information of each predicted entity, the entity local feature loss value is generated; Based on the entity relationship feature information of each sample and the entity relationship feature information of each predicted entity, an entity relationship feature loss value is generated; The entity feature loss value is obtained based on the entity local feature loss value and the entity relation feature loss value.

13. The method of claim 10, wherein obtaining a text feature loss value based on the sample entity text feature information and the predicted entity text feature information includes: Based on the sample entity text feature information, obtain the sample entity category text sub-information and entity category weight sub-information corresponding to each sample entity image, and obtain the predicted entity category text sub-information corresponding to each sample entity image based on the predicted entity text feature information. The sample entity category text sub-information is a semantic description of the instance association corresponding to the sample entity image, and the entity category weight sub-information represents the credibility of the sample entity category text sub-information. Based on the sample entity category text sub-information, predicted entity category text sub-information, and entity category weight sub-information corresponding to each sample entity image, a text feature loss value is generated.

14. A target detection method, comprising: Acquire the image to be detected; The image to be detected is input into the target detection model to obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model, wherein the target detection model is trained by the steps of the method according to any one of claims 1-13; Based on the entity image information of each sample and the entity category information corresponding to each entity image, target detection results are generated.

15. A target detection method, applied to a cloud device, comprising: A target detection request sent by a receiving end-side device, wherein the target detection request includes an image to be detected; The image to be detected is input into the target detection model to obtain at least one sample entity image information and entity category information corresponding to each entity image information generated by the target detection model, wherein the target detection model is trained by the steps of the method according to any one of claims 1-13; Based on the entity image information of each sample and the entity category information corresponding to each entity image, a target detection result is generated and the target detection result is sent to the edge device.

16. An electronic device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-15.

17. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-15.

18. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-15.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-15.

Images