Natural scene object detection method, system, device storage medium

Abstract

This invention discloses a method, system, and storage medium for object detection in natural scenes. It is a fine-tuning training scheme for a large visual language model for object detection tasks in natural scenes. Utilizing context learning strategies and fine-tuning techniques, it ensures high accuracy in text detection in natural scenes while exhibiting faster training speed and lower server resource consumption. A visual feature and text feature interleaving technique is proposed, combined with a context learning strategy, enabling the large language model to fully understand and effectively interact with textual and visual information. Based on contextual information, the model fully learns the relationship between the objects described in the text and the image, obtaining more accurate prediction results. Furthermore, supervised training is employed to effectively combine visual and textual information, improving the model's prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision language multimodal, and in particular to a method, system, device, and storage medium for object detection in natural scenes. Background Technology

[0002] In recent years, large-scale language modeling technology has developed rapidly. Through its application in the field of vision, visual-language large-scale models have also been extensively studied and applied in various fields, including knowledge-based question answering, robotics, and autonomous driving. Compared to single-modal information, visual-language large-scale models process data from two modalities simultaneously, obtaining richer semantic information. For detection in natural scenes, guided by textual information, they can achieve greater accuracy and flexibility.

[0003] Previous large-scale visual language models, when used for object detection tasks, could not output the precise location of text-related objects, such as coordinates, or required extensive pre-training. While pre-training methods can perform well in such tasks, they consume significant server resources and have long training times, making them unsuitable for widespread application. Previously, context learning strategies, also known as cue learning, were commonly used in large-scale language models. These models can learn relevant tasks and output content based on samples provided by the context, without requiring further training. Based on this, there have been applications in large-scale visual language models. One method has been developed for classification tasks using context learning strategies, but there has been no research on object detection tasks in natural scenes.

[0004] In summary, although existing methods have explored the performance of context learning strategies on downstream tasks in large visual language models, there has been no breakthrough in object detection tasks in natural scenes. These unexplored directions and unresolved issues provide direction for further research. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, device, and storage medium for object detection in natural scenes. The invention uses a context learning strategy for fine-tuning training for object detection tasks in natural scenes, thereby improving the output accuracy of the model in object detection tasks.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A method for object detection in natural scenes includes:

[0008] A dataset containing the location information of objects in a natural scene is obtained. The dataset is then reprocessed and reorganized according to a context learning strategy to obtain a training dataset. Each item in the training dataset contains multiple images, as well as the question text and label corresponding to each image.

[0009] A natural scene object detection model is constructed based on a large visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a large language model;

[0010] The natural scene object detection model is trained using a training dataset, including: extracting visual features from the images in each data set using a visual feature extraction module; extracting text features from the question text and labels corresponding to the first to second-to-last images in each data set, as well as the question text in the last image, using a text feature extraction module; interleaving visual and text features through a text and visual feature cross-layer; predicting the object location information in the last image of each data set using a large language model; calculating the loss function by combining the predicted object location information in the last image of each data set with the corresponding label; and training the natural scene object detection model.

[0011] Input the image and the corresponding question text into the trained natural scene object detection model to obtain the object location information in the image.

[0012] A natural scene object detection system, comprising:

[0013] The data acquisition and processing unit is used to acquire a dataset of natural scene object location information, and reprocess and organize the dataset according to a context learning strategy to obtain a training dataset; wherein, each data item in the training dataset contains multiple images, as well as the question text and label corresponding to each image;

[0014] The model building unit is used to build a natural scene object detection model based on a large visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer, and a large language model;

[0015] The model training unit is used to train the natural scene object detection model using the training dataset. This includes: extracting visual features from the images in each data set using a visual feature extraction module; extracting text features from the question text and labels corresponding to the first to second-to-last images in each data set, as well as the question text in the last image, using a text feature extraction module; interleaving visual and text features through a text and visual feature cross-layer; predicting the object location information in the last image of each data set using a large language model; combining the predicted object location information in the last image of each data set with the corresponding label; calculating the loss function; and training the natural scene object detection model.

[0016] The object detection unit is used to input the image and the corresponding question text into the trained natural scene object detection model to obtain the location information of objects in the image.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by the present invention, a fine-tuning training scheme for a large visual language model for object detection tasks in natural scenes is designed. Utilizing context learning strategies and fine-tuning techniques, it ensures high accuracy in text detection in natural scenes while exhibiting faster training speed and lower server resource consumption. A visual feature and text feature interleaving technique is proposed, combined with a context learning strategy, enabling the large language model to fully understand and effectively interact with textual and visual information. Based on contextual information, the model fully learns the connection between the text and the objects described in the image, obtaining more accurate prediction results. Furthermore, supervised training is employed to effectively combine visual and textual information, improving the model's prediction accuracy. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a natural scene object detection method provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of a dataset construction scheme provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the working process of the natural scene object detection model provided in an embodiment of the present invention;

[0025] Figure 4 The prediction effect diagram is provided for the embodiments of the present invention;

[0026] Figure 5 This is a schematic diagram of a natural scene object detection system provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0029] First, the following explanations are provided for the terms that may be used in this article:

[0030] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0031] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0032] The following provides a detailed description of a natural scene object detection method, system, device, and storage medium provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0033] Example 1

[0034] This invention provides a method for object detection in natural scenes, such as... Figure 1 As shown, it mainly includes the following steps:

[0035] Step 1: Obtain a dataset of natural scene object location information, and reprocess and organize the dataset according to the context learning strategy to obtain a training dataset.

[0036] In this embodiment of the invention, each data item in the training dataset contains multiple images, as well as question text and labels corresponding to each image.

[0037] The preferred implementation method for this step is as follows:

[0038] (1) Obtain a dataset of natural scene object location information, each data item containing at least an image and a corresponding label, the label containing the actual object location information in the image.

[0039] (2) Select T images and their corresponding labels, represented as: Y = {y1, y2, ..., y T}, X={x1,x2,...,x T}, using tags and based on the selected language model, generate the corresponding question text, represented as: Q={q1,q2,...,q T}; where each y represents an image, each x represents the label corresponding to the image, and each q represents the question text corresponding to the label. The subscript is the image number, and Y, X, and Q represent the sets consisting of images, labels, and question texts, respectively. Finally, Y, X, and Q constitute one data item in the training dataset. In one data item, the first T-1 images are samples for the context learning strategy. The natural scene object detection model uses the first T-1 images and their question texts and labels, as well as the Tth image and its question text, to predict the object location information in the Tth image, and the label of the Tth image provides supervision.

[0040] Then, an image-text interleaving strategy was adopted to interleave the question text, images, and labels of the data.

[0041] In this embodiment of the invention, the language model that generates the question text is generally a dedicated language model (e.g., GPT), which is used to provide a template for the question in order to improve the generalization of the dataset.

[0042] (3) The data is continuously reprocessed and reorganized in the manner described in step (2) above, and finally, a training dataset consisting of multiple data sets is obtained.

[0043] Step 2: Construct a natural scene object detection model based on the visual language big data model.

[0044] In this embodiment of the invention, the natural scene object detection model mainly includes: a visual feature extraction module, a text feature extraction module, a text and visual feature cross-layer, and a large language model.

[0045] Step 3: Train the natural scene object detection model using the training dataset.

[0046] The training process is as follows: Visual features are extracted from the images in each data set using a visual feature extraction module; text features are extracted from the question text and labels corresponding to the first to second-to-last images in each data set, as well as the question text in the last image, using a text feature extraction module; visual and text features are interleaved using a text and visual feature cross-layer; then, the object location information in the last image of each data set is predicted using a large language model; combining the predicted object location information in the last image of each data set with the corresponding label, a loss function is calculated, and the natural scene object detection model is trained.

[0047] Step 4: Input the image and the corresponding question text into the trained natural scene object detection model to obtain the object location information in the image.

[0048] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0049] I. Overview of the Plan.

[0050] This invention fills the gap in existing methods for fine-tuning large visual language models for natural scene object detection tasks. It proposes a context-learning-based fine-tuning training scheme for large visual language models (i.e., natural scene object detection models) used for natural scene object detection. For the first time, a context-learning strategy is used to fine-tune a visual language model specifically for natural scene object detection, thereby improving the model's output accuracy. Compared to pre-training schemes, the proposed fine-tuning strategy consumes fewer server resources, has a shorter training time, and is more suitable for applications in various scenarios and with different requirements. This technology not only achieves a significant improvement in visual effects but also reaches industry-leading levels in multiple quantitative performance metrics.

[0051] II. Detailed introduction of the plan.

[0052] 1. Data collection and processing.

[0053] In this embodiment of the invention, multiple open-source datasets providing location information of objects in natural scenes are obtained, and the data are reprocessed and reorganized according to a context learning strategy.

[0054] (1) For the images and labels in the dataset, take T images as a group. For example, if T=5, denoted as Y={y1,y2,y3,y4,y5}, where each y represents an image and the subscript is the image number.

[0055] Meanwhile, the labels of the original dataset are processed. Taking T=5 as an example, the label set of the images is denoted as X={x1,x2,x3,x4,x5}, where each x represents the label corresponding to the image.

[0056] (2) Process the annotation information of each image. Generate question text with similar meaning but different forms for each image through the open source language model. Here, we take T=5 as an example and denote it as Q={q1,q2,q3,q4,q5}. Each pair of questions and original image labels are combined as a data pair and denoteed as e1={q1,x1}. e1 represents the label pair corresponding to image y1. The label pairs of other images have a similar structure. The only difference in the representation is the subscript.

[0057] In this embodiment of the invention, the annotation information is the label mentioned above, which includes the object category in the image and the position coordinate information of each object. Through the selected language model, the required question text can be generated to form label pairs.

[0058] Therefore, after processing multiple open-source datasets, a new dataset D with a context learning strategy is formed. new Taking one data point in the dataset as an example, the data is organized in the form of D. t ={(q1,x1),(q2,x2),(q3,x3),(q4,x4),(q5,x5)}, such as Figure 2 As shown in Figure D. t Represented as the new dataset D new The t-th data item contains 5 original images from the original dataset. The first 4 images serve as samples for the context learning strategy. The large visual language model needs to be based on D... t The sample in the image and the question q5 in the 5th image are used to predict the location information of the corresponding object in y5, and then the loss function is calculated with the label x5 for training.

[0059] (3) When fine-tuning the model using the new dataset, data processing is performed beforehand. A text-image interleaving strategy is adopted to interleave the label information and image information of each data item in the dataset. When the language model of the visual language large model is trained, the text-image interleaving data format enables the model to fully perceive the contextual information of each data item in the dataset, giving full play to the advantages of the context learning strategy. The text-image interleaving data information organization structure is shown below: <q1> <y1> <x1> <q2> <y2> <x2> <q3> <y3> <x3> <q4> <y4> <q5> <y5>.

[0060] q represents dataset D new The problem is with the t-th data item in dataset D, where y represents the dataset D. new The image of the t-th data item in dataset D, where x represents the dataset D. new The image of the t-th data item corresponds to the answer to the question q. The data format is [X1,Y1,X2,Y2], which represents the position coordinates of the corresponding object in the image. The subscript is the image number, and the subscripts of q, y, and x are corresponding.

[0061] 2. Construct a large-scale visual language model framework and train it.

[0062] In this embodiment of the invention, a natural scene object detection model (i.e., a visual language large model framework) is constructed based on a visual language large model, which mainly includes: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer, and a language large model.

[0063] (1) Visual feature extraction module.

[0064] In this embodiment of the invention, the visual feature extraction module includes a visual converter and a linear projection layer.

[0065] The visual converter is adopted from the visual language large model (contrastive language image pre-trained model) to extract visual features from the images in each data item, represented as:

[0066]

[0067] in, Let y represent the image of the t-th data item in the training dataset. Each y represents an image, the superscript t indicates that the image belongs to the t-th data item in the training dataset, the subscript is the image index, and T is the number of images; ViT represents the visual converter, f t This represents the visual feature extracted from the image of the t-th data item in the training dataset.

[0068] Specifically, the visual converter is an open-source visual feature extraction model pre-trained on a large number of natural images. Based on the self-attention mechanism, it divides the input image into flat image blocks, and then inputs these image blocks as a sequence into the converter model to achieve visual feature extraction and processing, thereby obtaining visual features through the above process.

[0069] A linear projection layer is used to project visual features to the same dimension as text features in order to obtain a consistent representation in multimodal tasks, and is represented as follows:

[0070] f t p =W projector f t

[0071] Among them, W projector Let f be the parameter matrix of the linear projection layer. t p The projected visual features (i.e., visual features aligned with text features) serve as the input to the text-visual feature intersection layer.

[0072] Specifically, assuming the dimension of visual features belongs to d v Representing visual features f t Dimensions Let the notation be the set of real numbers, then the parameter matrix of the linear projection layer is... d t The dimension f represents the text feature. t p It is represented as a visual feature aligned with the text features after linear projection.

[0073] (2) Text feature extraction module.

[0074] In this embodiment of the invention, the text feature extraction module adopts the text feature extraction module in the visual language large model, and the process of extracting text features is represented as follows:

[0075]

[0076] Here, each q represents a question text, each x represents a label, the superscript t indicates that the question text and label belong to the t-th data item in the training dataset, the subscript is the index of the corresponding image, and T is the number of images; tokenizer represents the text feature extraction module, p t This represents the text feature extracted from the t-th item in the training dataset.

[0077] (3) Cross layer of text and visual features.

[0078] In this embodiment of the invention, the text and visual feature cross-layer simultaneously inputs the visual features and text features of the training data, and, based on a pre-designed text and image cross-structure, adds symbols to the text features to indicate image information. Visual features f t p With output text features p t Feature union is performed, inserting visual features into specified positions of text features, as shown below:

[0079]

[0080] Among them, F t cat p represents the data features after feature interleaving. t,tag This indicates the location of image markers in the text features. The above formula is used to add visual features to the corresponding positions in the text features, thus completing the interleaving of visual and text features. This represents the visual feature of the i-th image in the t-th data item, where T represents the number of images in a data item, i.e., the number mentioned earlier.

[0081]

[0082] In this embodiment of the invention, the symbol This involves inserting image features into the corresponding positions of text features, with the text containing a special token. , Figure 3 The data examples provide relevant explanations. A tag represents a corresponding identifier; image features are inserted sequentially into text features. The location completes the image and text feature interleaving.

[0083] In this embodiment of the invention, the designed text-image intersection structure is <image feature 1, text feature 1, image feature 2, text feature 2>. Figure 2 It also provides relevant cross-structure examples. The existing alignment of text features and image features is usually in the order of <image features, text features>. If there are multiple image inputs, the order changes to <image feature 1, image feature 2, text feature 1, text feature 2>. Through practice and related papers, it has been proven that the cross-structure designed in this invention is conducive to the full integration of text features and image features. (4) Language large model (pre-trained language large model).

[0084] In this embodiment of the invention, the language big model is based on a deep neural network converter and a long short-term memory network structure to process long-distance dependencies and capture complex patterns in the text. After pre-training with a large amount of data, it can generate natural and fluent text information.

[0085] In summary, this invention uses data features generated by the intersection of visual and textual features through a cross-layer of text and visual features to fine-tune a pre-trained large language model, allowing it to fully perceive the input visual and textual information, and guides the model to output the positional information of the corresponding object through context learning.

[0086] For the t-th data item in the training dataset, the language large model prediction output is represented as:

[0087]

[0088] Where T represents the number of images in a data item, i is the image index, p(.|.) represents the conditional probability, and the generated result is the text data with the highest probability, r i This represents the data features after feature interleaving corresponding to the (i-1)th image. Text data H has been generated i-1 Tag g t Text data generated under certain conditions; Text data H has been generated i-1 =r1,...,r i-1 When i-1 = 0, the corresponding data is empty; t represents the t-th data item in the training dataset. To determine the label for the last image in the t-th data item of the training dataset; express The predicted probability, The coordinates of the two focal points of the rectangle represent the position information of the object.

[0089] according to Determine the object location information in the last image of the t-th data item in the training dataset.

[0090] like Figure 3 The diagram shown illustrates the working process of a natural scene object detection model.

[0091] (5) Calculate the loss function.

[0092] Using the predicted and the labels included in the training dataset The loss function is calculated and expressed as:

[0093]

[0094] Among them, L d Let M be the loss function, and M represent the number of data items in the training dataset. This represents the object location information in the last image of the t-th data item in the predicted training dataset. The label is the last image corresponding to the t-th data item in the training dataset.

[0095] In this embodiment of the invention, gradient descent is used to train the entire model. During training, the model parameters of the visual converter and text feature extraction module are frozen and do not participate in training; the parameters of the linear projection layer and the large language model are unfrozen and participate in training. The loss function L is calculated... d And update the model parameters when the number of training iterations reaches a set number or the loss function L... d When convergence occurs, training stops, resulting in the trained natural scene object detection model (called the optimal detection model).

[0096] III. Object position detection.

[0097] After training, the trained natural scene object detection model can be used for object location detection tasks. Input the image and corresponding question text into the trained natural scene object detection model to obtain object location information in the image. For example... Figure 4 As shown, an example of the model detection results is provided; Figure 4 In this template, the first four items are the template and the last item is the problem. The purpose of this template is to guide the large visual language model to be trained. This is a method of fine-tuning the large model. The amount of data is much smaller than the amount of data used for pre-training the large model (generally, the pre-training data for large models is in the hundreds of millions). This method improves data utilization efficiency and achieves the desired effect with a small amount of data.

[0098] The above-mentioned solution provided by the embodiments of the present invention uses a context learning strategy to fine-tune the training of the model, so as to accurately identify the location of objects in natural scenes through context learning and text prompts. Compared with the pre-training scheme, the fine-tuning training strategy proposed in this invention consumes less server resources, has a shorter training time, and is more suitable for applications under various scenarios and requirements. This technology not only achieves a significant improvement in visual effects, but also reaches the industry-leading level in multiple quantitative performance indicators.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0100] Example 2

[0101] This invention also provides a natural scene object detection system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 5 As shown, the system mainly includes:

[0102] The data acquisition and processing unit is used to acquire a dataset of natural scene object location information, and reprocess and organize the dataset according to a context learning strategy to obtain a training dataset; wherein, each data item in the training dataset contains multiple images, as well as the question text and label corresponding to each image;

[0103] The model building unit is used to build a natural scene object detection model based on a large visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer, and a large language model;

[0104] The model training unit is used to train the natural scene object detection model using the training dataset. This includes: extracting visual features from the images in each data set using a visual feature extraction module; extracting text features from the question text and labels corresponding to the first to second-to-last images in each data set, as well as the question text in the last image, using a text feature extraction module; interleaving visual and text features through a text and visual feature cross-layer; predicting the object location information in the last image of each data set using a large language model; combining the predicted object location information in the last image of each data set with the corresponding label; calculating the loss function; and training the natural scene object detection model.

[0105] The object detection unit is used to input the image and the corresponding question text into the trained natural scene object detection model to obtain the location information of objects in the image.

[0106] Since the main technical details involved in this system have been described in detail in previous embodiments, they will not be repeated here.

[0107] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0108] Example 3

[0109] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0110] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0111] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0112] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0113] The output device can be a display terminal;

[0114] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0115] Example 4

[0116] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0117] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0118] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. < / q5> < / y4> < / q4> < / x3> < / y3> < / q3> < / x2> < / y2> < / q2> < / x1> < / y1> < / q1>

Claims

1. A method for object detection in natural scenes, characterized in that, include: A dataset containing the location information of objects in a natural scene is obtained. The dataset is then reprocessed and reorganized according to a context learning strategy to obtain a training dataset. Each item in the training dataset contains multiple images, as well as the question text and label corresponding to each image. A natural scene object detection model is constructed based on a large visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a large language model; The natural scene object detection model is trained using a training dataset, including: extracting visual features from the images in each data set using a visual feature extraction module; extracting text features from the question text and labels corresponding to the first to second-to-last images in each data set, as well as the question text in the last image, using a text feature extraction module; interleaving visual and text features through a text and visual feature cross-layer; predicting the object location information in the last image of each data set using a large language model; calculating the loss function by combining the predicted object location information in the last image of each data set with the corresponding label; and training the natural scene object detection model. Input the image and the corresponding question text into the trained natural scene object detection model to obtain the object location information in the image; The dataset for obtaining the location information of objects in the natural scene is reprocessed and reorganized according to a context learning strategy to obtain a training dataset, including: A dataset for obtaining the location information of objects in a natural scene, wherein each data item contains at least an image and a corresponding label, and the label contains the actual location information of the object in the image; Select T images and their corresponding labels, represented as: Y = {y1, y2, ..., y...} T }, X={x1,x2,...,x T }, using tags and based on the selected language model, generate the corresponding question text, represented as: Q={q1,q2,...,q T }; where each y represents an image, each x represents the label corresponding to the image, and each q represents the question text corresponding to the label. The subscript is the image number, and Y, X, and Q represent the sets consisting of images, labels, and question texts, respectively. Finally, Y, X, and Q constitute one data item in the training dataset. In one data item, the first T-1 images are samples for the context learning strategy. The natural scene object detection model uses the first T-1 images and their question texts and labels, as well as the Tth image and its question text, to predict the object location information in the Tth image, and the label of the Tth image provides supervision. A text-image interleaving strategy is used to interleave the question text, images, and labels in the data. By continuously reprocessing and reorganizing the data, a training dataset consisting of multiple data sets is eventually obtained. The method of interleaving visual features and text features through a text and visual feature cross-layer includes: The symbols added to the text features to indicate image information will transform the visual features f t p With text features p t Feature union is performed, inserting visual features into specified positions of text features, as shown below: Among them, F t cat This represents the data features after feature interleaving. p represents the visual feature of the i-th image in the t-th data item, T represents the number of images in a data item, and p t,tag This indicates the location where image tags are added to text features. The above formula is used to add image features to the corresponding positions in text features to complete the interleaving of visual and textual features.

2. The method for object detection in a natural scene according to claim 1, characterized in that, The visual feature extraction module includes: a visual converter and a linear projection layer; The visual converter, adopted from the visual language large model, is used to extract visual features from the image of each data item, represented as follows: in, Let y represent the image of the t-th data item in the training dataset. Each y represents an image, the superscript t indicates that the image belongs to the t-th data item in the training dataset, the subscript is the image index, and T is the number of images; ViT represents the visual converter, f t This represents the visual features extracted from the image of the t-th data item in the training dataset; A linear projection layer, used to project visual features onto the same dimension as text features, is represented as: f t p =W projector f t Among them, W projector Let f be the parameter matrix of the linear projection layer. t p The projected visual features serve as the input to the text-visual feature intersection layer.

3. The method for object detection in a natural scene according to claim 1, characterized in that, The text feature extraction module adopts the text feature extraction module in the visual language large model. The process of extracting text features is represented as follows: Here, each q represents a question text, each x represents a label, the superscript t indicates that the question text and label belong to the t-th data item in the training dataset, the subscript is the index of the corresponding image, and T is the number of images; tokenizer represents the text feature extraction module, p t This represents the text feature extracted from the t-th item in the training dataset.

4. The method for object detection in a natural scene according to claim 1, characterized in that, The object location information in the last image of each data item predicted by the language large model includes: For the t-th data item in the training dataset, the language large model prediction output is represented as: Where T represents the number of images in a data item, i is the image index, p(.|.) represents the conditional probability, and the generated result is the text data with the highest probability, r i This represents the data features after feature interleaving corresponding to the (i-1)th image. Text data H has been generated i-1 Tag g t Text data generated under certain conditions; t represents the t-th data item in the training dataset. To determine the label for the last image in the t-th data item of the training dataset; express The predicted probability, The coordinates of the two focal points of the rectangle represent the position information of the object. according to Determine the object location information in the last image of the t-th data item in the training dataset.

5. A method for detecting objects in a natural scene according to claim 1 or 4, characterized in that, The constructed loss function is expressed as: Among them, L d Let M be the loss function, and M represent the number of data items in the training dataset. This represents the object location information in the last image of the t-th data item in the predicted training dataset. The label is the last image corresponding to the t-th data item in the training dataset.

6. A natural scene object detection system, characterized in that, The method for implementing any one of claims 1 to 5 comprises: The data acquisition and processing unit is used to acquire a dataset of natural scene object location information, and reprocess and organize the dataset according to a context learning strategy to obtain a training dataset; wherein, each data item in the training dataset contains multiple images, as well as the question text and label corresponding to each image; The model building unit is used to build a natural scene object detection model based on a large visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer, and a large language model; The model training unit is used to train the natural scene object detection model using the training dataset. This includes: extracting visual features from the images in each data set using a visual feature extraction module; extracting text features from the question text and labels corresponding to the first to second-to-last images in each data set, as well as the question text in the last image, using a text feature extraction module; interleaving visual and text features through a text and visual feature cross-layer; predicting the object location information in the last image of each data set using a large language model; combining the predicted object location information in the last image of each data set with the corresponding label; calculating the loss function; and training the natural scene object detection model. The object detection unit is used to input the image and the corresponding question text into the trained natural scene object detection model to obtain the location information of objects in the image.

7. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images