Methods, apparatuses, and electronic devices for text-based image detection

Abstract

This application discloses a method, apparatus, and electronic device for text-based image detection. The method includes: acquiring target text and a target image; inputting the target text and target image into a target model, wherein the target model includes an encoder, a query module, and a decoder, the encoder being used at least to extract features from the target image and target text, the query module being used at least to determine the similarity between the extracted text features and image features, and the decoder being used at least to interact with the text features and image features based on a cross-attention mechanism; and determining the detection result through the target model. This application solves the technical problem of high detection cost and low detection efficiency caused by the need for separate modeling for each application scenario when performing object detection in different application scenarios in the prior art.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and more specifically, to a method, apparatus, and electronic device for detecting images based on text. Background Technology

[0002] With the advancement of technology, target detection technology is playing an increasingly important role in many fields such as land monitoring, water conservancy supervision, and environmental monitoring. For the detection needs of specific application scenarios, the traditional approach is to develop and deploy dedicated detection models for each application scenario. For example, water conservancy supervision application scenarios require a model specifically for identifying specific objects such as floods and dam cracks; while environmental monitoring application scenarios require another model to monitor forest fires, water pollution, etc.

[0003] However, while this strategy of developing customized models for a single scenario can provide a certain level of detection accuracy for specific objects, its limitations are becoming increasingly apparent. First, it is difficult to share dedicated models developed for different application scenarios. The generalization ability of the built dedicated models is insufficient, and existing models cannot be directly reused when adding new tasks. Model design and training must be carried out from scratch, resulting in repeated investment of resources and wasted development time. Second, developing dedicated models for each application scenario not only requires a lot of time and computing resources during the model training phase, but also faces high cost pressures during the model maintenance, update, and deployment phases. Especially when facing rapidly changing application scenarios and constantly updated detection requirements, it will further lead to technical problems of high detection costs and low detection efficiency.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for detecting images based on text, in order to at least solve the technical problems of high detection cost and low detection efficiency caused by the need to model each application scenario separately when performing object detection in different application scenarios in the prior art.

[0006] According to one aspect of this application, a method for detecting images based on text is provided, comprising: acquiring target text and a target image, wherein the target text is used to describe an object to be detected in the target image using a preset language, and the target image is a scene image corresponding to any one of E application scenarios, where E is a positive integer; inputting the target text and the target image into a target model, wherein the target model includes an encoder, a query module, and a decoder, wherein the encoder is used at least to extract features from the target image and the target text, the query module is used at least to determine the similarity between the extracted text features and image features, and the decoder is used at least to interact with the text features and image features based on a cross-attention mechanism; and determining a detection result through the target model, wherein the detection result is used to characterize the position of the object described by the target text in the target image.

[0007] Optionally, acquiring the target text and target image includes: periodically acquiring scene images corresponding to E application scenarios using E image acquisition devices to obtain an image dataset; using any image in the image dataset as the target image; and performing data cleaning on the original text corresponding to the target image to obtain the target text, wherein data cleaning is used to remove stop words from the original text, and the original text is used to describe objects that appear probably in the application scenario corresponding to the target image using a preset language.

[0008] Optionally, the detection result is determined by the target model, including: determining the semantic and contextual information included in the target text by the encoder in the target model, and using the semantic and contextual information included in the target text as the first text feature corresponding to the target text; extracting features from the target image by the encoder in the target model based on a multi-scale feature fusion strategy to obtain M first image features corresponding to the target image, where M is a positive integer; and determining the detection result based on the first text feature and the M first image features.

[0009] Optionally, determining the detection result based on the first text feature and M first image features includes: inputting the first text feature and M first image features into a query module in the target model; performing feature enhancement on the first text feature through the query module to obtain the second text feature, and performing feature enhancement on the M first image features to obtain the M second image features; obtaining a score matrix corresponding to the M second image features, wherein the scores in the score matrix are used to characterize the similarity between each second image feature and the second text feature; sorting the M second image features according to the scores in the score matrix to obtain an image feature sequence; using the top P second image features in the image feature sequence as P indices, wherein the P indices are used to determine the position of the object described by the target text in the target image, and P is a positive integer less than or equal to M; and determining the detection result based on the second text feature and the P indices.

[0010] Optionally, determining the detection result based on the second text feature and P indices includes: inputting the second text feature and P indices into the decoder in the target model; updating the second text feature through a text cross-attention layer in the decoder to obtain a third text feature, wherein the text cross-attention layer is used to enhance the second text feature; updating the P indices through an image cross-attention layer in the decoder to obtain P third image features, wherein the image cross-attention layer is used to enhance the P indices; and determining the detection result based on the similarity between each of the P third image features and the third text feature.

[0011] Optionally, the target model is trained through the following steps: preprocessing X open-source images obtained from the network to obtain Y open-source training images, where X and Y are both positive integers, and Y is greater than X. The preprocessing is used at least to perform random flipping, random cropping, random scaling, and normalization operations on the open-source images; using the Y open-source training images, Z bounding boxes, and W object types as the first dataset, where Z and W are both positive integers, and the W object types are described by text corresponding to a preset language; iteratively training the initial model based on the first dataset using a linear learning rate warm-up strategy and a learning rate jump strategy to obtain the first model; and determining the target model using the first model.

[0012] Optionally, determining the target model through the first model includes: preprocessing E historical scene images corresponding to E application scenarios to obtain U historical training images, where U is a positive integer and U is greater than E; using the U historical training images and the object types appearing in the historical training images as a second dataset; and iteratively training the first model based on the second dataset using a progressive learning rate decay strategy to obtain the target model.

[0013] According to another aspect of this application, an apparatus for detecting images based on text is also provided, comprising: an acquisition unit for acquiring target text and a target image, wherein the target text is used to describe an object to be detected in the target image using a preset language, and the target image is a scene image corresponding to any one of E application scenarios, where E is a positive integer; an input unit for inputting the target text and the target image into a target model, wherein the target model includes an encoder, a query module, and a decoder, the encoder is used at least to extract features from the target image and the target text, the query module is used at least to determine the similarity between the extracted text features and image features, and the decoder is used at least to interact with the text features and image features based on a cross-attention mechanism; and a determination unit for determining the detection result through the target model, wherein the detection result is used to characterize the position of the object described by the target text in the target image.

[0014] According to another aspect of this application, a computer program product is also provided, which stores a computer program, wherein, when the computer program is running, it controls the computer program product to perform any of the above-mentioned methods for detecting images based on text.

[0015] According to another aspect of this application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the text-based image detection method described above.

[0016] In this application, target text and target image are first obtained. The target text is used to describe the object to be detected in the target image using a preset language. The target image is a scene image corresponding to any one of E application scenarios, where E is a positive integer. Then, the target text and target image are input into a target model, which includes an encoder, a query module, and a decoder. The encoder is used at least to extract features from the target image and target text. The query module is used at least to determine the similarity between the extracted text features and image features. The decoder is used at least to interact with the text features and image features based on a cross-attention mechanism. Finally, the detection result is determined by the target model, where the detection result is used to characterize the position of the object described by the target text in the target image.

[0017] As can be seen from the above, this application pre-trains a target model, which includes an encoder, a query module, and a decoder. The decoder can interact with text features and image features based on a cross-attention mechanism, thereby achieving the goal of aligning text features and image features and overcoming the influence of image complexity from different scenarios. Therefore, the target model trained in this application can perform object detection on scene images corresponding to any of the E application scenarios, thus eliminating the need for separate modeling for different application scenarios and achieving the goal of reducing object detection costs and improving object detection efficiency.

[0018] Therefore, this application achieves the goal of avoiding separate modeling for different application scenarios of the target image source by using a target model to detect objects in the target image based on the target text. This results in reducing the cost of object detection and improving the efficiency of object detection. It also solves the technical problem of high detection cost and low detection efficiency caused by the need to model separately for each application scenario when performing object detection in different application scenarios in the prior art. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram of a dedicated model corresponding to a single optional application scenario;

[0021] Figure 2 This is a flowchart of an optional open-set target detection method;

[0022] Figure 3 This is a flowchart of an optional text-based image detection method according to an embodiment of this application;

[0023] Figure 4 This is a structural diagram of an optional detection system according to an embodiment of this application;

[0024] Figure 5 This is a flowchart of another optional text-based image detection method according to an embodiment of this application;

[0025] Figure 6 This is a flowchart of an optional target model according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of an optional text-based image detection apparatus according to an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] It should also be noted that all relevant information (including but not limited to information used for display and analysis) and data (including but not limited to image data, text data, and scene data) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it needs to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent from the aforementioned user or organization.

[0031] Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant information and data involved in this application all comply with the relevant laws, regulations, and standards of the relevant regions, and necessary confidentiality measures have been taken. This application does not violate public order and good morals. In addition, this application provides a corresponding operation entry point for users to choose to agree to or refuse authorization. If the user chooses to refuse authorization, the corresponding expert decision-making process will be initiated.

[0032] In one alternative embodiment, a dedicated model is provided for detection in a single application scenario, such as engineering construction, smoke and fire detection, or people swimming. Figure 1 This is a schematic diagram of a dedicated model corresponding to a single optional application scenario. The training methods for the dedicated model include:

[0033] (1) Data collection and processing: Collect data from specific real-world scenarios and perform preprocessing.

[0034] (2) Model training: Design the training process, modify the model structure and parameters, and complete the model training according to the specific characteristics of the data in the specific scenario.

[0035] (3) Model deployment: The models trained for different application scenarios will be deployed separately.

[0036] In one alternative embodiment, an open-set target detection method is provided. Figure 2 This is a flowchart of an optional open-set object detection method, such as... Figure 2 As shown, the method includes:

[0037] (1) Data preprocessing: The target detection and phrase localization tasks are pre-trained in the same way so that the data of the two tasks can complement each other and thus jointly improve the zero-shot target detection capability of the model. In the data preprocessing stage, the datasets of the two tasks are unified into the same form of text input and image input. The text input is the name of the object to be identified using a period, and the image input is the image of target detection or phrase localization.

[0038] (2) Encoding different modal data: A dual-tower structure is adopted, and the input data is encoded by a pre-trained text encoder and an image encoder respectively to obtain the encoded text features and visual features.

[0039] (3) Cross-modal feature fusion and enhancement: Text features and visual features are interacted through a cross-attention mechanism to achieve deep fusion of multiple modalities.

[0040] (4) Feature similarity calculation: Calculate the similarity between the image region features and the text prompt word features, match the regions with high similarity with the corresponding text, and obtain the detection results.

[0041] However, the technical solutions corresponding to the above two embodiments have the following shortcomings:

[0042] (1) Existing dedicated model development solutions for multi-scenario detection tasks require each task to develop a separate model, which results in high development costs, low efficiency, and difficulty in meeting the needs of complex multi-scenario tasks.

[0043] (2) Most existing open set object detection models only support English text input. When applied in Chinese environment, additional translation steps are required, which not only increases the time cost, but also affects the detection accuracy due to translation errors.

[0044] According to an embodiment of this application, an embodiment of a method for detecting images based on text is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] This application provides a system for detecting images based on text (hereinafter referred to as the detection system) to perform the method for detecting images based on text in this application. Figure 3 This is a flowchart of an optional text-based image detection method according to an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps:

[0046] Step S301: Obtain the target text and target image.

[0047] In step S301, the target text is used to describe the object to be detected in the target image using a preset language. The target image is a scene image corresponding to any one of the E application scenarios, where E is a positive integer.

[0048] Optionally, the above E application scenarios include at least: land monitoring, water conservancy supervision, and environmental monitoring scenarios.

[0049] Optionally, the preset language is Chinese.

[0050] Optionally, the target images can be acquired by cameras installed at medium to high points in E application scenarios, thereby ensuring that the acquired target images include scene data corresponding to a larger monitoring area in the application scenarios.

[0051] Step S302: Input the target text and target image into the target model.

[0052] In step S302, the target model includes an encoder, a query module, and a decoder. The encoder is used at least to extract features from the target image and the target text. The query module is used at least to determine the similarity between the extracted text features and image features. The decoder is used at least to interact with the text features and image features based on a cross-attention mechanism.

[0053] In this embodiment, the target text can be described by a preset language, which can be set to Chinese. That is, the target model in this application can support Chinese recognition. In other words, the encoder in this application supports feature extraction of the target text described by the preset language, omitting the process of translating Chinese into English required by the model created in the prior art, thereby avoiding the introduction of errors in the translation process and improving the accuracy of object detection of the target image.

[0054] In addition, the decoder in this embodiment uses a cross-attention mechanism to enable in-depth interaction between text features and image features. The cross-attention mechanism helps the target model to comprehensively consider the visual cues included in the target image and the semantic information corresponding to the target text during the detection process, thereby improving the detection accuracy of the target model.

[0055] Step S303: Determine the detection result using the target model.

[0056] In step S303, the detection results are used to characterize the position of the object described in the target text in the target image.

[0057] Optionally, the detection system marks the position of the object described by the target text in the target image as an anchor box, and returns the annotation result as the detection result to the user or relevant technical personnel, so that the user or relevant technical personnel can intuitively see the actual position of the object corresponding to the target text through the image, thereby improving the user experience.

[0058] Furthermore, this application also allows users to view the detection results via a browser. When a user needs to perform object detection on an application scene, they first log in to the browser corresponding to the detection system, input the target text, target image, and confidence level. The confidence level setting directly affects the accuracy of the detection results. Although a lower confidence level threshold can detect more targets, it can also increase the probability of false detections. Subsequently, this application can also determine the category of the object corresponding to the target text through the target model, output the category of the object as a label, and compare the label with the label set included in the application scene corresponding to the target image, thereby judging whether the application scene classification of the target image is correct based on the comparison results.

[0059] As can be seen from the above, this application pre-trains a target model, which includes an encoder, a query module, and a decoder. The decoder can interact with text features and image features based on a cross-attention mechanism, thereby achieving the goal of aligning text features and image features and overcoming the influence of image complexity from different scenarios. Therefore, the target model trained in this application can perform object detection on scene images corresponding to any of the E application scenarios, thus eliminating the need for separate modeling for different application scenarios and achieving the goal of reducing object detection costs and improving object detection efficiency.

[0060] Therefore, this application achieves the goal of avoiding separate modeling for different application scenarios of the target image source by using a target model to detect objects in the target image based on the target text. This results in reducing the cost of object detection and improving the efficiency of object detection. It also solves the technical problem of high detection cost and low detection efficiency caused by the need to model separately for each application scenario when performing object detection in different application scenarios in the prior art.

[0061] In one optional embodiment, the detection system first periodically acquires scene images corresponding to E application scenarios using E image acquisition devices to obtain an image dataset; then, the detection system uses any one image from the image dataset as the target image; next, the detection system performs data cleaning on the original text corresponding to the target image to obtain the target text, wherein the data cleaning is used to remove stop words from the original text, and the original text is used to describe objects that appear probably in the application scenario corresponding to the target image using a preset language.

[0062] Optionally, Figure 4 This is a structural diagram of an optional detection system according to an embodiment of this application, such as... Figure 4 As shown, the detection system deploys E mid-to-high-point image acquisition devices in E application scenarios, such as smoke detection, construction, and swimming. The data management server centrally stores the image datasets acquired by the E mid-to-high-point image acquisition devices in a data storage server. When object detection is required, the data management server retrieves the data according to user needs. After processing, the detected image data is transmitted as the target image to a high-performance computing server (i.e., the target model). The high-performance computing server is a mid-to-high-point multi-scenario open-set target detection model that supports Chinese input. Scene detection is performed using the target model. After an anomaly is detected, the detection result is input to the alarm management system. Finally, the alarm management system inputs the alarm information to a monitoring center display screen that allows interaction with users or technicians.

[0063] Optionally, the E high-point image acquisition devices can also acquire images of their respective monitoring areas at preset time intervals, such as once every 10 minutes, to ensure that image data that fully reflects scene changes can be collected.

[0064] Optionally, when the target scenario is an engineering construction application scenario, the objects described in the target text are construction equipment such as excavators, trucks, or tower cranes.

[0065] In the above embodiments, this application cleans the original text corresponding to the target image by removing semantically meaningless stop words and other noise information, making the target text input to the target model more refined. Therefore, the target text can directly point to the target object that needs to be focused on in the image, thereby reducing the redundant computation of the target model when processing the target text, accelerating the speed of feature extraction and matching of the target model, and thus improving the overall efficiency of the target model in object detection.

[0066] In addition, the cleaned target text focuses on key semantic information, avoiding incorrect training directions caused by noise in the original text. This allows the target model to understand the text description more accurately, enabling it to correctly locate and identify objects in the target image, thereby improving the accuracy of the detection results output by the target model.

[0067] In one optional embodiment, the detection system determines the semantic and contextual information included in the target text through the encoder in the target model, and uses the semantic and contextual information included in the target text as the first text feature corresponding to the target text. Then, the detection system extracts features from the target image based on a multi-scale feature fusion strategy through the encoder in the target model to obtain M first image features corresponding to the target image, where M is a positive integer. Then, the detection system determines the detection result based on the first text feature and the M first image features.

[0068] Optionally, the encoder in the target model includes a text encoder and a visual encoder, wherein the text encoder is an encoder that supports preset language text prompts.

[0069] For example, when the preset language is Chinese, the target model uses BERT (Bidirectional Encoder Representations from Transformers) trained on Chinese corpus to process the target text and generate the first text features. The text encoder includes a 12-layer Transformer structure, with a hidden layer dimension of 768 and 12 attention heads. After the target text is input, it is first segmented by the word segmenter in the text encoder. Then, it is converted into a vector representation through the word embedding layer in the text encoder. Finally, it is used for feature extraction through the multi-layer Transformer structure in the text encoder, thereby achieving the purpose of capturing the semantic information and contextual relationships of the Chinese text.

[0070] For example, the detection system uses a Swin-Transformer (window multi-head self-attention transformer) as the visual encoder. The structure of this visual encoder includes four stages, each containing 2, 2, 6, and 2 Transformer structures respectively. The feature dimension captured by the visual encoder starts from 96 and doubles in each stage, eventually reaching 768. The number of heads in the multi-head attention is [3, 6, 12, 24]. Furthermore, the visual encoder adopts a multi-scale feature fusion strategy, thereby improving the accuracy of detecting objects of different sizes in the target image.

[0071] In one optional embodiment, the detection system first inputs the first text feature and M first image features into the query module in the target model. Then, the detection system enhances the first text feature through the query module to obtain the second text feature, and enhances the M first image features to obtain the M second image features. Then, the detection system obtains the score matrix corresponding to the M second image features, wherein the score in the score matrix is ​​used to characterize the similarity between each second image feature and the second text feature.

[0072] In addition, the detection system sorts the M second image features according to the scores in the score matrix to obtain an image feature sequence. Then, the detection system uses the top P second image features in the image feature sequence as P indices, where the P indices are used to determine the position of the object described by the target text in the target image, and P is a positive integer less than or equal to M. Finally, the detection system determines the detection result based on the second text features and the P indices.

[0073] Optionally, the formula for calculating the fractional matrix is ​​shown in the following formula (1).

[0074] (1)

[0075] In the above formula (1), L is a fractional matrix. For the second image feature, 'b' represents the first text feature, and 'b' represents the batch size. The number of tokens for the image. d is the number of tokens in the text, i is the i-th sample in the batch, j is the j-th token in the image features, k is the k-th token in the text features, and l is the l-th dimension in the feature dimensions.

[0076] Optionally, the token mentioned above is a fixed-length feature vector converted from each of the multiple small blocks obtained by segmenting an image or text.

[0077] Then, the maximum score corresponding to each image token is as shown in the following formula (2):

[0078] (2)

[0079] Finally, as shown in the following formula (3), the P second image features with the highest scores are selected as the P indices.

[0080] (3)

[0081] In the above embodiments, the query module can enhance text features through a general self-attention mechanism and enhance image features through a deformable self-attention mechanism, thereby obtaining second text features and M second image features. This achieves the goal of strengthening the robustness and discriminative power of text and image feature representations based on the self-attention mechanism, enabling text features to better summarize and describe the object, and image features to more accurately capture object details, thereby improving the detection accuracy of the target model.

[0082] In the above embodiments, each score in the score matrix represents the matching score between a second image feature and a second text feature. The higher the value, the stronger the correlation between the two. The detection system sorts the M second image features from high to low according to the obtained score matrix. Then, the target model can perform a preliminary query based on the sorting results, that is, select the top P second image features as P indices. The P indices are used to indicate the position of the object described by the target text in the target image. Finally, the detection system performs precise detection, that is, by comprehensively analyzing the second text features and the P indices, it further determines the accurate position of the object described by the target text. In the above recognition process, this application utilizes the guidance of the second text features and the pointing of the P indices to achieve the purpose of narrowing the detection range of precise detection, thereby improving the detection accuracy of the target model.

[0083] In one optional embodiment, the detection system inputs a second text feature and P indices into a decoder in the target model. Then, the detection system updates the second text feature through a text cross-attention layer in the decoder to obtain a third text feature, wherein the text cross-attention layer is used to enhance the second text feature. Next, the detection system updates the P indices through an image cross-attention layer in the decoder to obtain P third image features, wherein the image cross-attention layer is used to enhance the P indices. Finally, the detection system determines the detection result based on the similarity between each of the P third image features and the third text feature.

[0084] Optionally, the text cross-attention layer is as shown in the following formula (4):

[0085] (4)

[0086] Optionally, the image cross-attention layer is as shown in the following formula (5):

[0087] (5)

[0088] In the above formulas (4) and (5), Q represents the query vector, K represents the key vector, and V represents the feature vector that needs to be weighted and summed by the attention mechanism.

[0089] In the above embodiments, the text cross-attention layer allows the target model to incorporate semantic information of the text when processing images. This not only deepens the target model's understanding of the semantics of the target text, but also enables image features to be adjusted according to the guidance of the target text. This allows the target model to more accurately capture image regions that match the description of the target text. The image cross-attention layer, on the other hand, enables the first text features to absorb corresponding visual cues from the image, thereby enhancing its ability to represent specific scenes and objects. This bidirectional feature enhancement process promotes a deeper interaction between the target text and the target image, thereby improving the accuracy of the target model's detection.

[0090] In addition, objects in different application scenarios are allowed to have similar visual appearances but completely different semantics. By introducing text cross-attention layers and image cross-attention layers, the detection system enables the target model to take into account both visual similarity and semantic differences during detection. This allows the target model to adapt quickly in various application scenarios. Even when faced with unseen detection objects or application scenarios, it can still perform detection based on existing knowledge and text descriptions, thereby improving the generalization performance of the target model.

[0091] In one optional embodiment, the training steps of the target model include: First, the detection system preprocesses X open-source images obtained from the network to obtain Y open-source training images, where X and Y are both positive integers, and Y is greater than X. The preprocessing is used at least to perform random flipping, random cropping, random scaling, and normalization operations on the open-source images. Then, the detection system uses the Y open-source training images, Z bounding boxes, and W object types as a first dataset, where Z and W are both positive integers, and the W object types are described by text corresponding to a preset language. Then, the detection system iteratively trains the initial model based on the first dataset using a linear learning rate warm-up strategy and a learning rate jump strategy to obtain a first model. Finally, the detection system determines the target model based on the first model.

[0092] In the above embodiments, the linear learning rate warm-up strategy accelerates model convergence by gradually increasing the learning rate in the early stages of the model. This helps the model explore the solution space more quickly and avoids problems such as slow training, vanishing gradients, or exploding gradients caused by improper initial learning rate settings. Furthermore, the warm-up strategy can smoothly start the training process, enabling the model to quickly adapt to the data distribution in the first dataset and shorten the training time.

[0093] In addition, by using a learning rate jump strategy, which involves reducing the learning rate after a certain number of training rounds, it helps to prevent the model from overfitting the training data in the later stages of training. The detection system can adjust the weights more finely with a lower learning rate, thereby reducing the overreaction to noisy data. This allows the model to maintain its complexity while improving its generalization ability, enabling the target model to maintain good detection performance on unseen Chinese and image data, and thus improving the detection accuracy of the trained model.

[0094] Furthermore, the first dataset in this application is a large-scale sample set. The detection system is initially trained on the large-scale sample set, which enables the trained model to learn the rich expressions of the Chinese language. Combined with linear warm-up and learning rate jump strategies, the features learned by the model at multiple stages will be richer, thereby improving the robustness of the trained model.

[0095] In an optional embodiment, the training steps of the target model further include: first, the detection system preprocesses E historical scene images corresponding to E application scenarios to obtain U historical training images, where U is a positive integer and U is greater than E; then, the detection system uses the U historical training images and the object types appearing in the historical training images as a second dataset; finally, the detection system iteratively trains the first model based on the second dataset using a progressive learning rate decay strategy to obtain the target model.

[0096] In the above embodiment, the detection system iteratively trains the first model based on the second dataset using a progressive learning rate decay strategy. A relatively small learning rate is used during training, with the following effect:

[0097] (1) The progressive learning rate decay strategy ensures that the model can gradually adjust its weights during training, from the initial rapid learning of basic features to the final fine adjustment, making it more in line with the data distribution in the second dataset with small samples. This makes the target model more sensitive to target detection in specific scenarios, even when the amount of data is relatively limited, thereby improving the detection accuracy of the target model.

[0098] (2) Using a smaller learning rate for fine-tuning can limit the magnitude of weight updates during model training, thereby reducing the over-reliance on specific data features in the second dataset with small samples, thus improving the generalization ability of the target model.

[0099] (3) Fine-tuning training based on a small second dataset by progressively decaying the learning rate and using a smaller learning rate in the later stages of training can reduce unnecessary iterations, reduce the training cost of the target model, and thus improve the overall training efficiency.

[0100] As can be seen from the above, this application pre-trains a target model, which includes an encoder, a query module, and a decoder. The decoder can interact with text features and image features based on a cross-attention mechanism, thereby achieving the goal of aligning text features and image features and overcoming the influence of image complexity from different scenarios. Therefore, the target model trained in this application can perform object detection on scene images corresponding to any of the E application scenarios, thus eliminating the need for separate modeling for different application scenarios and achieving the goal of reducing object detection costs and improving object detection efficiency.

[0101] Therefore, this application achieves the goal of avoiding separate modeling for different application scenarios of the target image source by using a target model to detect objects in the target image based on the target text. This results in reducing the cost of object detection and improving the efficiency of object detection. It also solves the technical problem of high detection cost and low detection efficiency caused by the need to model separately for each application scenario when performing object detection in different application scenarios in the prior art.

[0102] In one alternative embodiment, Figure 5 This is a flowchart of another optional text-based image detection method according to an embodiment of this application, such as... Figure 5 As shown, the method includes: inputting Chinese text and the image to be detected into a target model; preprocessing the Chinese text and the image to be detected through the target model; then encoding the text and the image separately through the target model; then performing query selection guided by Chinese text; performing cross-modal decoding based on the selection result; and finally outputting the detection result.

[0103] In one alternative embodiment, Figure 6 This is a flowchart of an optional target model according to an embodiment of this application, such as... Figure 6 As shown, the target model structure includes: a text encoder, a visual encoder, a selection module (i.e., the query module mentioned above), and a decoder.

[0104] Optionally, after extracting text features and image features through the text encoder and visual encoder, this application inputs the text features and image features into a cross-modal feature fusion module for feature enhancement. This module contains multiple feature enhancer layers. The cross-modal feature fusion module uses deformable self-attention to enhance image features and ordinary self-attention to enhance text features. In addition, the target model also adds image-to-text cross-attention and text-to-image cross-attention modules to perform feature fusion on text features and image features, thereby aligning text features and image features. Then, the selection module selects image features that are more relevant to the input text as the query of the decoder, outputs 900 indices, and initializes the query based on the selected 900 indices.

[0105] Optionally, the decoder is a cross-modal decoder, which employs a 6-layer Transformer structure with a hidden layer dimension of 256 and 8 attention heads. During processing, each query first passes through a self-attention layer, then interacts with image and text features through text cross-attention layers and image cross-attention layers respectively. Next, the interaction results are fed into a feedforward neural network layer (intermediate layer dimension of 1024) for further query processing, thereby improving the feature representation capability. Finally, the location of the object described in the text is determined by calculating the similarity between text and image features.

[0106] In the above embodiments, this design enables cross-modal decoders to establish closer connections between different modalities, thereby improving the performance of the target model in visual and language understanding tasks.

[0107] Optionally, the target model in this application includes two training phases: pre-training and fine-tuning. The pre-training phase is a crucial step for the model to learn general features. The pre-training process uses a large-scale object detection dataset containing 365 common categories, 2 million images, and 30 million bounding boxes. The pre-training phase enables the model to learn object detection capabilities in diverse scenes, providing a foundation for general recognition capabilities in subsequent fine-tuning. The specific data processing flow for pre-training is as follows:

[0108] (1) Data preprocessing: including random flipping, random cropping, random scaling and normalization techniques. Among them, random flipping technology flips the image horizontally with a probability of 0.5, which increases the diversity of training data. Random cropping technology is to crop out part of the image to simulate the situation of target occlusion. Random scaling technology is to perform target detection at different scales, thereby enhancing the performance of the model at different resolutions. Normalization technology normalizes the pixel values ​​of the image to meet the input requirements of the pre-trained model.

[0109] (2) The optimization strategies in the pre-training stage include: using the AdamW (Adaptive Moment Estimation with Decoupled Weight Decay) optimizer, with an initial learning rate of 0.0004, adopting a linear learning rate warm-up strategy (i.e., gradually increasing to the set value in the first 1000 steps) and a learning rate jump strategy (i.e., reducing the learning rate by 10 times in the 19th and 26th rounds), using weight decay to prevent overfitting, and limiting the maximum gradient norm to 0.1 by gradient clipping.

[0110] (3) In the pre-training stage, each batch of training uses 144 images, and distributed training is performed on multiple graphics processors using 144 images, thereby accelerating the training process and improving the training efficiency of the model.

[0111] (4) The loss functions in the pre-training stage include: the first loss function and the second loss function. The first loss function is used for classification loss, which can handle the problem of imbalanced sample classes and ensure the detection effect of small class targets. The second loss function is used for bounding box regression loss, thereby ensuring that the bounding box of the target can match the real location and improve the detection accuracy.

[0112] (5) During the pre-training phase, the English labels included in the large-scale object detection dataset were converted to Chinese. Since there are semantic differences between English and Chinese, the labels need to be checked a second time during the pre-training phase to ensure that the labeled objects in the dataset correspond to the Chinese labels in a semantically accurate manner, thereby avoiding the reduction of the detection effect of the target model in Chinese scenes due to translation errors.

[0113] Optionally, the fine-tuning stage is largely similar to the pre-training stage. The fine-tuning stage uses historical images sampled from different application scenarios and their corresponding annotations to create a fine-tuning dataset. A smaller learning rate is used in the fine-tuning stage to avoid destroying the general features learned in the pre-training stage. Simultaneously, for domain-specific terminology, the fine-tuning stage appropriately transforms and adjusts the labels corresponding to these terms to ensure that the target model can accurately understand and detect industry-specific targets. The specific data processing flow for fine-tuning is as follows:

[0114] (1) Fine-tuning dataset construction: During the fine-tuning stage, images sampled from different scenarios (such as illegal mining, smoke and fire detection, dust monitoring, etc.) were labeled to construct a fine-tuning dataset for a specific domain. These datasets cover typical applications of object detection in various industries, ensuring that the model can meet the specific detection needs of the industry.

[0115] (2) Sample preprocessing: Similar to the pre-training stage, it includes random flipping, random cropping and random scaling techniques.

[0116] (3) Learning rate adjustment: In the fine-tuning stage, a smaller learning rate was adopted to avoid destroying the general features learned in the pre-training. The learning rate was gradually reduced through a progressive learning rate decay strategy, so that the model could be stably optimized on the fine-tuning dataset and better adapt to the specific goals of the application scenario.

[0117] (4) Label Conversion and Adjustment: Labels were appropriately converted and adjusted to suit the specific terminology and expressions used in the relevant field. To ensure the model can accurately understand and detect industry-specific targets, the label conversion process specifically considered field-specific terminology and underwent detailed checks and adjustments. This process ensures the model can accurately understand industry-specific labels during target detection.

[0118] In the above embodiments, the role of multi-scene object detection using the target model is as follows:

[0119] (1) The target model has general target detection capabilities and supports multiple downstream tasks, such as engineering construction, smoke and fire identification, dust monitoring, etc. It is no longer necessary to develop a dedicated model for each task, which improves the versatility of the target model.

[0120] (2) The target model structure includes a text encoder that supports Chinese prompts and provides a complete Chinese training process, which solves the problem that existing technologies can only handle English input, avoids the loss and time overhead caused by translation, and thus improves the detection efficiency in Chinese scenes.

[0121] (3) The target model adopts a training process of pre-training and fine-tuning, which not only ensures the generality of the model, but also optimizes the application scenarios in specific fields during the fine-tuning process, thereby improving the detection accuracy and application effect.

[0122] According to another aspect of the embodiments of this application, an apparatus for detecting images based on text is also provided. Figure 7 This is a schematic diagram of an optional text-based image detection apparatus according to an embodiment of this application, such as... Figure 7 As shown, the device for detecting images based on text includes: an acquisition unit 701, an input unit 702, and a determination unit 703.

[0123] Optionally, the acquisition unit is used to acquire target text and target image, wherein the target text is used to describe the object to be detected in the target image using a preset language, and the target image is a scene image corresponding to any one of E application scenarios, where E is a positive integer; the input unit is used to input the target text and target image into the target model, wherein the target model includes an encoder, a query module, and a decoder, wherein the encoder is used at least to extract features from the target image and target text, the query module is used at least to determine the similarity between the extracted text features and image features, and the decoder is used at least to interact with the text features and image features based on a cross-attention mechanism; the determination unit is used to determine the detection result through the target model, wherein the detection result is used to characterize the position of the object described by the target text in the target image.

[0124] In one optional embodiment, the acquisition unit includes: a timed acquisition subunit, a first determination subunit, and a data cleaning subunit.

[0125] Optionally, the timed acquisition subunit is used to acquire scene images corresponding to E application scenarios through E image acquisition devices at regular intervals to obtain an image dataset; the first determination subunit is used to take any image in the image dataset as the target image; the data cleaning subunit is used to clean the original text corresponding to the target image to obtain the target text, wherein the data cleaning is used to remove stop words in the original text, and the original text is used to describe objects that appear probably in the application scenario corresponding to the target image through a preset language.

[0126] In one optional embodiment, the determining unit includes: a second determining subunit, a feature extraction subunit, and a third determining subunit.

[0127] Optionally, the second determining subunit is used to determine the semantic and contextual information included in the target text through the encoder in the target model, and use the semantic and contextual information included in the target text as the first text feature corresponding to the target text; the feature extraction subunit is used to extract features from the target image through the encoder in the target model based on a multi-scale feature fusion strategy, and obtain M first image features corresponding to the target image, where M is a positive integer; the third determining subunit is used to determine the detection result based on the first text feature and the M first image features.

[0128] In one optional embodiment, the third determining subunit includes: an input module, a feature enhancement module, an acquisition module, a sorting module, a first determining module, and a second determining module.

[0129] Optionally, the input module is used to input the first text features and M first image features into the query module of the target model; the feature enhancement module is used to enhance the first text features through the query module to obtain the second text features, and to enhance the M first image features to obtain the M second image features; the acquisition module is used to acquire the score matrix corresponding to the M second image features, wherein the scores in the score matrix are used to characterize the similarity between each second image feature and the second text features; the sorting module is used to sort the M second image features according to the scores in the score matrix to obtain an image feature sequence; the first determination module is used to use the top P second image features in the image feature sequence as P indices, wherein the P indices are used to determine the position of the object described by the target text in the target image, and P is a positive integer less than or equal to M; the second determination module is used to determine the detection result based on the second text features and the P indices.

[0130] In one optional embodiment, the second determining module includes: an input submodule, a first update submodule, a second update submodule, and a determining submodule.

[0131] Optionally, an input submodule is used to input the second text features and P indices into the decoder in the target model; a first update submodule is used to update the second text features through a text cross-attention layer in the decoder to obtain third text features, wherein the text cross-attention layer is used to enhance the features of the second text features; a second update submodule is used to update the P indices through an image cross-attention layer in the decoder to obtain P third image features, wherein the image cross-attention layer is used to enhance the features of the P indices; and a determination submodule is used to determine the detection result based on the similarity between each of the P third image features and the third text features.

[0132] In one optional embodiment, the apparatus for detecting images based on text further includes: a first preprocessing unit, a first dataset determination unit, a first iterative training unit, and a model determination unit.

[0133] Optionally, the first preprocessing unit is used to preprocess X open-source images obtained from the network to obtain Y open-source training images, where X and Y are both positive integers, and Y is greater than X. The preprocessing is used to perform random flipping, random cropping, random scaling, and normalization operations on the open-source images. The first dataset determination unit is used to take the Y open-source training images, Z bounding boxes, and W object types as the first dataset, where Z and W are both positive integers, and the W object types are described by text corresponding to a preset language. The first iterative training unit is used to iteratively train the initial model based on the first dataset using a linear learning rate warm-up strategy and a learning rate jump strategy to obtain a first model. The model determination unit is used to determine the target model based on the first model.

[0134] In one optional embodiment, the device for detecting images based on text further includes: a second preprocessing unit, a second dataset determination unit, and a second iterative training unit.

[0135] Optionally, the second preprocessing unit is used to preprocess the E historical scene images corresponding to the E application scenarios to obtain U historical training images, where U is a positive integer and U is greater than E; the second dataset determination unit is used to take the U historical training images and the object types appearing in the historical training images as the second dataset; the second iterative training unit is used to iteratively train the first model based on the second dataset using a progressive learning rate decay strategy to obtain the target model.

[0136] As can be seen from the above, this application pre-trains a target model, which includes an encoder, a query module, and a decoder. The decoder can interact with text features and image features based on a cross-attention mechanism, thereby achieving the goal of aligning text features and image features and overcoming the influence of image complexity from different scenarios. Therefore, the target model trained in this application can perform object detection on scene images corresponding to any of the E application scenarios, thus eliminating the need for separate modeling for different application scenarios and achieving the goal of reducing object detection costs and improving object detection efficiency.

[0137] In addition, the encoder in this application supports feature extraction of target text described in a preset language, omitting the process of translating the preset language into English required by existing models, thereby avoiding errors introduced during the translation process and improving the accuracy of object detection in images.

[0138] Therefore, this application achieves the goal of avoiding separate modeling for different application scenarios of the target image source by using a target model to detect objects in the target image based on the target text. This results in reducing the cost of object detection and improving the efficiency of object detection. It also solves the technical problem of high detection cost and low detection efficiency caused by the need to model separately for each application scenario when performing object detection in different application scenarios in the prior art.

[0139] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product including a stored computer program, wherein, when the computer program is running, the computer program product is controlled to perform any of the above-mentioned methods for detecting images based on text.

[0140] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the text-based image detection method described above by executing the executable instructions.

[0141] Optionally, Figure 8 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 8 As shown, this application provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-mentioned methods for detecting images based on text.

[0142] The above-described embodiments or examples disclosed in this application are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this application. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, various embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0143] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0144] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0148] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of detecting an image based on text, characterized by, The method comprises: obtaining target text and a target image, wherein the target text is used to describe an object to be detected in the target image by a preset language, and the target image is a scene image corresponding to any one of E application scenarios, and E is a positive integer; inputting the target text and the target image into a target model, wherein the target model comprises an encoder, a query module and a decoder, the encoder is used at least for feature extraction of the target image and the target text, the query module is used at least for determining the similarity between the extracted text features and image features, and the decoder is used at least for interaction between the text features and the image features based on a cross-attention mechanism, and training samples of the target model at least include Y open source training images, Z bounding boxes, W object types, U historical training images obtained by preprocessing historical scene images corresponding to the E application scenarios, and object types appearing in the U historical training images, Y, Z, W and U are positive integers; and determining a detection result by the target model, wherein the detection result is used to represent the position of the object described by the target text in the target image; wherein determining the detection result by the target model comprises: determining semantic information and context information included in the target text by the encoder in the target model, and taking the semantic information and the context information included in the target text as first text features corresponding to the target text; performing feature extraction on the target image based on a multi-scale feature fusion strategy by the encoder in the target model to obtain M first image features corresponding to the target image, wherein M is a positive integer; and determining the detection result according to the first text features and the M first image features; wherein determining the detection result according to the first text features and the M first image features comprises: inputting the first text features and the M first image features into the query module in the target model; performing feature enhancement on the first text features by the query module to obtain second text features, and performing feature enhancement on the M first image features to obtain M second image features; obtaining a score matrix corresponding to the M second image features, wherein the scores in the score matrix are used to represent the similarity between each second image feature and the second text features; sorting the M second image features according to the scores in the score matrix to obtain an image feature sequence; taking the top P second image features in the image feature sequence as P indexes, wherein the P indexes are used to determine the position of the object described by the target text in the target image, and P is a positive integer less than or equal to M; and determining the detection result according to the second text features and the P indexes. The determining the detection result according to the second text feature and the P indexes comprises: inputting the second text feature and the P indexes into a decoder in the target model; updating the second text feature through a text cross-attention layer in the decoder to obtain a third text feature, wherein the text cross-attention layer is used for feature enhancement on the second text feature; updating the P indexes through an image cross-attention layer in the decoder to obtain P third image features, wherein the image cross-attention layer is used for feature enhancement on the P indexes; and determining the detection result according to a similarity between each of the P third image features and the third text feature.

2. The method for detecting an image based on text according to claim 1, wherein, The target text and the target image are acquired, comprising: E image acquisition devices are used to collect scene images corresponding to the E application scenarios at regular time intervals to obtain an image dataset; Any image in the image dataset is taken as the target image; Data cleaning is performed on an original text corresponding to the target image to obtain the target text, wherein the data cleaning is used to remove stop words in the original text, and the original text is used to describe objects that are likely to appear in an application scenario corresponding to the target image in the preset language.

3. The method for detecting an image based on text according to claim 1, wherein, The target model is obtained through the following steps: X open source images obtained from a network are preprocessed to obtain Y open source training images, wherein X and Y are positive integers, Y is greater than X, and the preprocessing is at least used for random flipping, random cropping, random scaling and normalization operation on the open source images; The Y open source training images, Z boundary boxes and W object types are taken as a first dataset, wherein Z and W are positive integers, and the W object types are described by a text corresponding to the preset language; An initial model is iteratively trained based on the first dataset through a linear learning rate warm-up strategy and a learning rate jump strategy to obtain a first model; The target model is determined through the first model.

4. The method for detecting an image based on text according to claim 3, wherein, The target model is determined through the first model, comprising: E historical scene images corresponding to the E application scenarios are preprocessed to obtain U historical training images, wherein U is a positive integer and U is greater than E; The U historical training images and object types appearing in the historical training images are taken as a second dataset; The first model is iteratively trained based on the second dataset through a gradual learning rate decay strategy to obtain the target model.

5. An apparatus for detecting an image based on text, the apparatus comprising: Comprising: An acquisition unit is configured to acquire a target text and a target image, wherein the target text is used to describe an object to be detected in the target image in a preset language, and the target image is a scene image corresponding to any one of E application scenarios, and E is a positive integer. The input unit is configured to input the target text and the target image into a target model, wherein the target model comprises an encoder, a query module, and a decoder, the encoder is configured to perform feature extraction on at least the target image and the target text, the query module is configured to determine similarity between extracted text features and image features, and the decoder is configured to interact the text features and the image features based on a cross-attention mechanism, and training samples of the target model at least include Y open-source training images, Z bounding boxes, W object types, U historical training images obtained by preprocessing historical scene images corresponding to E application scenarios, and object types appearing in the U historical training images, Y, Z, W, and U are positive integers; The determination unit is configured to determine a detection result by using the target model, wherein the detection result is used to represent a position of an object described by the target text in the target image; The determination unit comprises: a second determination subunit configured to determine semantic information and context information included in the target text by using the encoder in the target model, and use the semantic information and the context information included in the target text as first text features corresponding to the target text; a feature extraction subunit configured to perform feature extraction on the target image based on a multi-scale feature fusion strategy by using the encoder in the target model, and obtain M first image features corresponding to the target image, wherein M is a positive integer; and a third determination subunit configured to determine the detection result according to the first text features and the M first image features. The third determination subunit comprises: an input module configured to input the first text features and the M first image features into the query module in the target model; a feature enhancement module configured to perform feature enhancement on the first text features by using the query module to obtain second text features, and perform feature enhancement on the M first image features to obtain M second image features; an acquisition module configured to acquire a score matrix corresponding to the M second image features, wherein a score in the score matrix is used to represent similarity between each second image feature and the second text features; a sorting module configured to sort the M second image features according to the scores in the score matrix to obtain an image feature sequence; a first determination module configured to use P second image features ranking at the front in the image feature sequence as P indexes, wherein the P indexes are used to determine the position of the object described by the target text in the target image, and P is a positive integer less than or equal to M; and a second determination module configured to determine the detection result according to the second text features and the P indexes. The second determining module comprises: an input submodule configured to input the second text feature and the P indexes into a decoder in the target model; a first updating submodule configured to update the second text feature by a text cross-attention layer in the decoder to obtain a third text feature, wherein the text cross-attention layer is configured to perform feature enhancement on the second text feature; a second updating submodule configured to update the P indexes by an image cross-attention layer in the decoder to obtain P third image features, wherein the image cross-attention layer is configured to perform feature enhancement on the P indexes; and a determining submodule configured to determine the detection result according to a similarity between each of the P third image features and the third text feature.

6. A computer program product, characterised in that, The computer program product comprises a computer program, wherein the computer program controls the computer program product to execute the method for detecting an image based on a text according to any one of claims 1 to 4 when the computer program is running.

7. An electronic device, comprising: The computer program product comprises a computer program, wherein the computer program controls the computer program product to execute the method for detecting an image based on a text according to any one of claims 1 to 4 when the computer program is running. The computer program product comprises a computer program, wherein the computer program controls the computer program product to execute the method for detecting an image based on a text according to any one of claims 1 to 4 when the computer program is running.

Images