Natural scene object detection method and system, and equipment storage medium
By using context learning strategies to fine-tune the visual language model in natural scenes, combining visual and text features, the problem of precise position output in natural scene object detection tasks is solved, achieving high accuracy and rapid training.
Patent Information
- Application Number
- CN202411051936.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The existing visual language model is difficult to output the exact location of text-related objects on object detection tasks in natural scenarios, and the pre-training method consumes a lot of server resources, has a long training time, and has no relevant context learning strategy research.
The context learning strategy is used to fine-tune the training of natural scene object detection tasks. Through the visual feature extraction module, the text feature extraction module and the text and visual feature intersection layer, the language model is combined with the supervision and training to improve the accuracy of the model's output on the object detection task. Spend.
It achieves high accuracy of object detection in natural scenarios, reduces server resource consumption and training time, is suitable for a variety of scenario applications, and reaches the industry-leading level in visual effects and quantitative performance.
Smart Images

Figure CN119007069A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal computer vision language, and in particular to a natural scene object detection method, system, and device storage medium. Background Art
[0002] In recent years, language big model technology has developed rapidly. Through its application in the field of vision, visual language big models have also been widely studied and applied in many fields, including knowledge question answering, robotics, and autonomous driving. Compared with single-modal information, visual language big models process data from two modalities at the same time to obtain richer semantic information. For detection in natural scenes, it can be more accurate and flexible under the guidance of text information.
[0003] Some previous large visual language models could not output the precise location of text-related objects, such as coordinate values, in object detection tasks, or required a large amount of data for pre-training. Although the pre-training method can also perform well on such tasks, it consumes a lot of server resources and takes a long time to train, and is not universally applicable. Previously, the contextual learning strategy, also known as prompt learning, was often used for tasks related to large language models. The model was able to learn related tasks and output content based on samples provided by the context without training. Based on this, there are also related applications on large visual language models. Previously, there was a method that used the contextual learning strategy to perform classification tasks on large visual language models, but there has been no related research on object detection tasks in natural scenes.
[0004] In summary, although there are existing methods that attempt to use contextual learning strategies on downstream tasks on large visual language models, there has been no breakthrough in object detection tasks in natural scenes. These unexplored directions and unresolved problems provide directions for further research. Summary of the invention
[0005] The purpose of the present invention is to provide a natural scene object detection method, system, and device storage medium, which use a context learning strategy to perform fine-tuning training for natural scene object detection tasks, thereby improving the output accuracy of the model on object detection tasks.
[0006] The objective of the present invention is achieved through the following technical solutions:
[0007] A natural scene object detection method, comprising:
[0008] Obtain a data set of natural scene object position information, and reprocess and organize the data set according to a context learning strategy to obtain a training data set; wherein each item of data in the training data set includes multiple images, and a question text and a label corresponding to each image;
[0009] Build a natural scene object detection model based on the visual language model, including: visual feature extraction module, text feature extraction module, text and visual feature cross layer and language model;
[0010] The natural scene object detection model is trained using a training data set, including: extracting visual features from images in each item of data through a visual feature extraction module, extracting text features from question texts and labels corresponding to the first to second to last images in each item of data, and the question text of the last image through a text feature extraction module; interleaving visual features with text features through a text and visual feature cross layer, and then predicting object position information in the last image of each item of data through a large language model; combining the predicted object position information in the last image of each item of data with the corresponding label, calculating a loss function, and training the natural scene object detection model;
[0011] Input the image and the corresponding question text to the trained natural scene object detection model to obtain the object location information in the image.
[0012] A natural scene object detection system, comprising:
[0013] A data acquisition and processing unit, used to acquire a data set of natural scene object position information, and reprocess and organize the data set according to a context learning strategy to obtain a training data set; wherein each item of data in the training data set includes multiple images, and a question text and a label corresponding to each image;
[0014] A model building unit, used to build a natural scene object detection model based on the visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a language model;
[0015] A model training unit, used for training the natural scene object detection model using a training data set, including: extracting visual features from images in each item of data through a visual feature extraction module, extracting text features from question texts and labels corresponding to the first to second to last images in each item of data, and the question text of the last image through a text feature extraction module; interleaving visual features with text features through a text and visual feature cross layer, and then predicting object position information in the last image of each item of data through a large language model; combining the predicted object position information in the last image of each item of data with the corresponding label, calculating a loss function, and training the natural scene object detection model;
[0016] The object detection unit is used to input the image and the corresponding question text into the trained natural scene object detection model to obtain the object position information in the image.
[0017] A processing device, comprising: one or more processors; a memory for storing one or more programs;
[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0019] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.
[0020] It can be seen from the technical solution provided by the present invention that a fine-tuning training scheme for a large visual language model for object detection tasks in natural scenes is designed. The context learning strategy and fine-tuning technology are used to ensure the accuracy of text detection in natural scenes while showing faster training speed and less server resource consumption. A visual feature and text feature interleaving technology is proposed, combined with a context learning strategy, so that the large language model can fully understand text information and visual information and interact effectively. According to the context, the model fully learns the connection between the text and the objects described in the image to obtain more accurate prediction results. In addition, supervised training is used for training to effectively combine visual information and text information, thereby improving the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0022] Figure 1 A flowchart of a natural scene object detection method provided by an embodiment of the present invention;
[0023] Figure 2 A schematic diagram of a data set construction scheme provided in an embodiment of the present invention;
[0024] Figure 3 A schematic diagram of the working process of the natural scene object detection model provided by an embodiment of the present invention;
[0025] Figure 4 A prediction effect diagram provided by an embodiment of the present invention;
[0026] Figure 5 A schematic diagram of a natural scene object detection system provided by an embodiment of the present invention;
[0027] Figure 6 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.
[0029] First, the terms that may be used in this article are explained as follows:
[0030] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.
[0031] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.
[0032] The following is a detailed description of a natural scene object detection method, system, and device storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer shall be followed.
[0033] Embodiment 1
[0034] The embodiment of the present invention provides a natural scene object detection method, such as Figure 1 As shown, it mainly includes the following steps:
[0035] Step 1: Obtain a dataset of natural scene object position information, reprocess and organize the dataset according to the context learning strategy, and obtain a training dataset.
[0036] In the embodiment of the present invention, each item of data in the training data set includes multiple images, and question text and labels corresponding to each image.
[0037] The preferred implementation of this step is as follows:
[0038] (1) Obtain a dataset of natural scene object location information, where each item of data at least includes an image and a corresponding label, and the label includes the actual object location information in the image.
[0039] (2) Select T images and their corresponding labels, expressed as: Y = {y 1 ,y 2 ,...,y T}, X={x 1 ,x 2 ,...,x T}, using the label and based on the selected language model to generate the corresponding question text, expressed as: Q = {q 1 ,q 2 ,...,q T}; where each y represents an image, each x represents the label corresponding to the image, and each q represents the question text corresponding to the label. The subscript is the image serial number. Y, X, and Q represent the set of images, labels, and question texts, respectively. Finally, Y, X, and Q constitute one item of data in the training data set. In one item of data, the first T-1 images are samples of the context learning strategy. The natural scene object detection model predicts the object location information in the Tth image through the first T-1 images and their question texts and labels, as well as the Tth image and its question text, and is supervised by the label of the Tth image.
[0040] Afterwards, the question text, image and label of the data are interlaced using the image-text interlacing strategy.
[0041] In the embodiment of the present invention, the language model for generating the question text is generally a dedicated language model (eg, GPT), which is used to provide a template for the question to improve the generalization of the data set.
[0042] (3) Continuously reprocess and organize the data in accordance with the method in the aforementioned step (2), and finally obtain a training data set consisting of multiple data.
[0043] Step 2: Build a natural scene object detection model based on the visual language model.
[0044] In the embodiment of the present invention, the natural scene object detection model mainly includes: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a language large model.
[0045] Step 3: Use the training data set to train the natural scene object detection model.
[0046] The training process is as follows: extract visual features from the images in each data item through the visual feature extraction module, and extract text features from the question texts and labels corresponding to the first to the second to last images in each data item, as well as the question text of the last image, through the text feature extraction module; interleave the visual features and text features through the text and visual feature cross layer, and then predict the object position information in the last image of each data item through the language large model; combine the predicted object position information in the last image of each data item with the corresponding label, calculate the loss function, and train the natural scene object detection model;
[0047] Step 4: Input the image and the corresponding question text into the trained natural scene object detection model to obtain the object location information in the image.
[0048] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.
[0049] 1. Overview of the plan.
[0050] The present invention fills the gap in the existing fine-tuning methods of large visual language models for natural scene object detection tasks, and proposes a fine-tuning training scheme for a large visual language model (i.e., a natural scene object detection model) for natural scene object detection based on a contextual learning strategy. For the first time, a contextual learning strategy is used to fine-tune the visual language model for natural scene object detection tasks, thereby improving the output accuracy of the model on object detection tasks. Compared with pre-training schemes, the fine-tuning strategy proposed in the present invention consumes less server resources, has a shorter training time, and is more suitable for applications in a variety of scenarios and requirements. This technology not only achieves significant improvements in visual effects, but also reaches the industry-leading level in multiple quantitative performance indicators.
[0051] 2. Detailed introduction of the plan.
[0052] 1. Data collection and processing.
[0053] In an embodiment of the present invention, a plurality of open source data sets providing position information of natural scene objects are obtained, and the data are reprocessed and organized according to a contextual learning strategy.
[0054] (1) For the images and labels in the dataset, we take T images as a group, and take T = 5 as an example, denoted as Y = {y 1 ,y 2 ,y 3 ,y 4 ,y5}, where each y represents an image and the subscript is the image number.
[0055] At the same time, the labels of the original data set are processed. Taking T = 5 as an example, the label set of the image is recorded as X = {x 1 ,x 2 ,x 3 ,x 4 ,x 5}, where each x represents the label corresponding to the image.
[0056] (2) Process the annotation information of each image and generate question text with similar meaning but different forms for each image through the open source language model. Here, we take T = 5 as an example and record it as Q = {q 1 ,q 2 ,q 3 ,q 4 ,q 5}, each pair of questions and original image labels is combined as a data pair, denoted as e 1 = {q 1 ,x 1},e 1 Represents image y 1 The corresponding label pairs,the label pairs of the remaining images have similar structures,,and the only difference in the representation is the subscript.
[0057] In the embodiment of the present invention, the annotation information is the label mentioned above, which includes the object category in the image and the position coordinate information of each object. Through the selected language model, the required question text can be generated to form a label pair.
[0058] Therefore, after processing multiple open source datasets, a new dataset D with contextual learning strategy is formed. new , taking one item in the data set as an example, the data organization form is D t ={(q 1 ,x 1 ),(q 2 ,x 2 ),(q 3 ,x 3 ),(q 4 ,x 4 ),(q 5 ,x 5 )},like Figure 2 As shown. t Denoted as a new dataset D new The t-th item of data in contains 5 original images in the original dataset. The first 4 images are used as samples of the context learning strategy. The visual language model needs to be based on D tThe sample in and the question q of the 5th image 5 , predict y 5 The location information of the corresponding object in the 5 Calculate the loss function for training.
[0059] (3) When fine-tuning the model using a new data set, data processing is performed in advance, and the image and text interleaving strategy is used to interleave the label information and image information of each item in the data set. When the data is output to the language model of the visual language model for training, the image and text interleaving data format enables the model to fully perceive the context information of each item in the data set, giving full play to the advantages of the context learning strategy. The information organization structure of the image and text interleaving data is as follows: 1 > <y 1 > <x 1 > 2 > <y 2 > <x 2 > 3 > <y 3 > <x 3 > 4 > <y 4 > 5 > <y 5 >.
[0060] q represents the data set D new The problem of the t-th data in , y represents the data set D new The image of the t-th data in, x represents the dataset D new The image of the t-th data in the answer to the corresponding question q, the data format is [X 1 ,Y 1 ,X 2 ,Y 2 ], which represents the position coordinate frame of the corresponding object in the image; the subscript is the sequence number of the image, and the subscripts of q, y, and x are corresponding.
[0061] 2. Build a large visual language model framework and train it.
[0062] In an embodiment of the present invention, a natural scene object detection model (i.e., a visual language big model framework) is constructed based on a visual language big model, which mainly includes: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a language big model.
[0063] (1) Visual feature extraction module.
[0064] In the embodiment of the present invention, the visual feature extraction module includes: a visual converter and a linear projection layer.
[0065] The visual converter uses the visual converter in the visual language large model (comparative language image pre-training model) to extract visual features from the image in each data item, expressed as:
[0066]
[0067] in, represents the image of the t-th item in the training dataset. Each y represents an image. The superscript t indicates that the image belongs to the t-th item in the training dataset. The subscript is the sequence number of the image, and T is the number of images. ViT represents the visual converter, and f t represents the visual features extracted from the image of the t-th data in the training dataset.
[0068] Specifically: The visual converter is an open source visual feature extraction model that has been pre-trained on a large number of natural images. Based on the self-attention mechanism, the input image is divided into flat image blocks, and then these image blocks are input into the converter model as a sequence to realize visual feature extraction and processing, and then the visual features are obtained through the above.
[0069] The linear projection layer is used to project the visual features to the same dimension as the text features in order to obtain consistent representation in multimodal tasks, expressed as:
[0070] f t p =W projector f t
[0071] Among them, W projector is the parameter matrix of the linear projection layer, f t p is the projected visual feature (i.e., the visual feature aligned with the text feature), which serves as the input of the cross-layer of text and visual features.
[0072] Specifically, assuming that the dimension of the visual feature belongs to d v Represents the visual feature f t The dimension of is a real number set symbol, then the parameter matrix of the linear projection layer is d t Represents the dimension of text features, f t p Represented as visual features aligned with text features after linear projection.
[0073] (2) Text feature extraction module.
[0074] In the embodiment of the present invention, the text feature extraction module adopts the text feature extraction module in the visual language model, and the process of extracting text features is expressed as follows:
[0075]
[0076] Among them, each q represents a question text, each x represents a label, the superscript t indicates that the question text and the label belong to the tth item in the training data set, the subscript is the serial number of the corresponding image, and T is the number of images; tokenizer represents the text feature extraction module, p t Represents the text features extracted from the t-th item of data in the training dataset.
[0077] (3) Cross-layer of text and visual features.
[0078] In the embodiment of the present invention, the text and visual feature cross layer simultaneously inputs the visual features and text features of the training data, and according to the pre-designed text and image cross structure, the symbols for marking image information added to the text features are The visual feature f t p And the output text feature p t Perform feature union and insert the visual features into the specified position of the text features, expressed as:
[0079]
[0080] Among them, F t cat represents the data features after feature interleaving, p t,tag Indicates the position of the image mark in the text feature. The visual feature is added to the corresponding position of the text feature through the above formula to complete the interlacing of visual and text features. represents the visual features of the i-th image in the t-th data item, and T represents the number of images contained in a data item, that is,
[0081]
[0082] In the embodiment of the present invention, the symbol The image features are inserted into the corresponding positions of the text features, and there will be special token marks in the text , Figure 3 There are related instructions in the data sample. Tag means the corresponding tag, insert the image features into the text features in order. The location where the image and text features are interleaved.
[0083] In the embodiment of the present invention, the designed text and image cross structure is <image feature 1, text feature 1, image feature 2, text feature 2>, Figure 2Related cross-structure examples are also provided. The existing alignment of text features and image features is usually in the order of <image feature, text feature>. If multiple images are input, the order is changed to <image feature 1, image feature 2, text feature 1, text feature 2>. Practice and related papers have proved that the cross-structure designed by the present invention is conducive to the full integration of text features and image features. (4) Language macro model (pre-trained language macro model).
[0084] In an embodiment of the present invention, the language model is based on a converter of a deep neural network and a long short-term memory network structure to process long-distance dependencies and capture complex patterns in texts. After pre-training with a large amount of data, natural and fluent text information can be generated.
[0085] The implementation summary of the present invention is to fine-tune the pre-trained language model through the data features after the visual text features are interleaved through the text and visual feature cross-layer, so that it can fully perceive the input visual information and text information, and guide the model to output the location information of the corresponding object through context learning;
[0086] For the t-th item in the training data set, the language model prediction output is expressed as:
[0087]
[0088] Among them, T represents the number of images contained in a data item, i is the sequence number of the image, p(.|.) represents the conditional probability, and the generated result is the text data with the highest probability, r i Represents the data features after the features corresponding to the i-1th image are interleaved Generated text data H i-1 , label g t Text data generated under certain conditions; Generated text data H i-1 =r 1 ,...,r i-1 ; When i-1 = 0, the corresponding data is empty data; t represents the tth item of data in the training data set, is the label corresponding to the last image of the t-th item in the training dataset; express The predicted probability of are the position coordinates of the two focus points of the rectangular frame, and the rectangular frame is the object position information;
[0089] according to Determine the object location information in the last image of the t-th item in the training dataset
[0090] like Figure 3As shown, this is a schematic diagram of the working process of the natural scene object detection model.
[0091] (5) Calculate the loss function.
[0092] Using the predicted and the labels contained in the training dataset Calculate the loss function, expressed as:
[0093]
[0094] Among them, L d is the loss function, M means that the training data set contains M items of data, Represents the predicted object location information in the last image of the t-th item in the training dataset. It is the label corresponding to the last image of the t-th item in the training dataset.
[0095] In the embodiment of the present invention, the gradient descent method is used to train the entire model, wherein during the training process, the model parameters of the visual converter and the text feature extraction module are frozen and do not participate in the training; the parameters of the linear projection layer and the language model are unfrozen and participate in the training. By calculating the loss function L d And update the model parameters, when the number of training iterations reaches the set number or the loss function L d When convergence occurs, the training stops, thereby obtaining a trained natural scene object detection model (called the optimal detection model).
[0096] 3. Object position detection.
[0097] After training, the trained natural scene object detection model can be used to perform object position detection tasks. The image and the corresponding question text are input to the trained natural scene object detection model to obtain the object position information in the image. Figure 4 As shown, an example of the model detection results is provided; Figure 4 In the example, the first four items are templates, and the last item is a question. The purpose of this is to guide the large visual language model to be trained in the form of this template. This is a method of fine-tuning the large model. The amount of data is much smaller than the amount of data for pre-training of the large model (generally, the pre-training data of the large model is in the hundreds of millions). This method is used to improve data utilization efficiency and achieve ideal results using a small amount of data.
[0098] The above-mentioned scheme provided by the embodiment of the present invention uses a context learning strategy to fine-tune the model for training, so as to realize accurate identification of object positions in natural scenes through context learning and text prompts. Compared with the pre-training scheme, the fine-tuning training strategy proposed by the present invention consumes less server resources and has a shorter training time, and is more suitable for applications in various scenarios and requirements. This technology not only achieves significant improvement in visual effects, but also reaches the industry-leading level in multiple quantitative performance indicators.
[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0100] Embodiment 2
[0101] The present invention also provides a natural scene object detection system, which is mainly used to implement the method provided in the above embodiment, such as Figure 5 As shown, the system mainly includes:
[0102] A data acquisition and processing unit, used to acquire a data set of natural scene object position information, and reprocess and organize the data set according to a context learning strategy to obtain a training data set; wherein each item of data in the training data set includes multiple images, and a question text and a label corresponding to each image;
[0103] A model building unit, used to build a natural scene object detection model based on the visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a language model;
[0104] A model training unit, used for training the natural scene object detection model using a training data set, including: extracting visual features from images in each item of data through a visual feature extraction module, extracting text features from question texts and labels corresponding to the first to second to last images in each item of data, and the question text of the last image through a text feature extraction module; interleaving visual features with text features through a text and visual feature cross layer, and then predicting object position information in the last image of each item of data through a large language model; combining the predicted object position information in the last image of each item of data with the corresponding label, calculating a loss function, and training the natural scene object detection model;
[0105] The object detection unit is used to input the image and the corresponding question text into the trained natural scene object detection model to obtain the object position information in the image.
[0106] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.
[0107] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.
[0108] Embodiment 3
[0109] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.
[0110] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.
[0111] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:
[0112] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;
[0113] The output device may be a display terminal;
[0114] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.
[0115] Embodiment 4
[0116] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.
[0117] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.
[0118] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A natural scene object detection method, characterized in that: include: Obtain a data set of natural scene object position information, and reprocess and organize the data set according to a context learning strategy to obtain a training data set; wherein each item of data in the training data set includes multiple images, and a question text and a label corresponding to each image; Build a natural scene object detection model based on the visual language model, including: visual feature extraction module, text feature extraction module, text and visual feature cross layer and language model; The natural scene object detection model is trained using a training data set, including: extracting visual features from images in each item of data through a visual feature extraction module, extracting text features from question texts and labels corresponding to the first to second to last images in each item of data, and the question text of the last image through a text feature extraction module; interleaving visual features with text features through a text and visual feature cross layer, and then predicting object position information in the last image of each item of data through a large language model; combining the predicted object position information in the last image of each item of data with the corresponding label, calculating a loss function, and training the natural scene object detection model; Input the image and the corresponding question text to the trained natural scene object detection model to obtain the object location information in the image.
2. A natural scene object detection method according to claim 1, characterized in that: The dataset of obtaining the position information of the natural scene objects, and reprocessing and organizing the dataset according to the context learning strategy to obtain the training dataset comprises: Obtain a data set of natural scene object position information, each item of data at least including an image and a corresponding label, and the label includes the real object position information in the image; Select T images and their corresponding labels, expressed as: Y = {y1, y2, ..., y T }, X={x1,x2,...,x T }, using the labels and based on the selected language model to generate the corresponding question text, expressed as: Q = {q1, q2, ..., q T }; where each y represents an image, each x represents the label corresponding to the image, and each q represents the question text corresponding to the label. The subscript is the image serial number. Y, X, and Q represent the set of images, labels, and question texts, respectively. Finally, Y, X, and Q constitute one item of data in the training data set. In one item of data, the first T-1 images are samples of the context learning strategy. The natural scene object detection model predicts the object location information in the Tth image through the first T-1 images and their question texts and labels, as well as the Tth image and its question text, and is supervised by the label of the Tth image. The problem text, images and labels of the data are interlaced by using the graphic-text interlacing strategy; The data is continuously reprocessed and organized, and finally, a training data set consisting of multiple data is obtained.
3. A natural scene object detection method according to claim 1 or 2, characterized in that: The visual feature extraction module includes: a visual converter and a linear projection layer; Among them, the visual converter uses the visual converter in the visual language model to extract visual features from the image in each data item, which is expressed as: in, represents the image of the t-th item in the training dataset. Each y represents an image. The superscript t indicates that the image belongs to the t-th item in the training dataset. The subscript is the sequence number of the image, and T is the number of images. ViT represents the visual converter, and f t represents the visual features extracted from the image of the t-th data in the training dataset; The linear projection layer is used to project the visual features to the same dimension as the text features, expressed as: f t p =W projector f t Among them, W projector is the parameter matrix of the linear projection layer, f t p is the projected visual feature, which serves as the input of the cross-layer of text and visual features.
4. A natural scene object detection method according to claim 1 or 2, characterized in that: The text feature extraction module adopts the text feature extraction module in the visual language model, and the process of extracting text features is expressed as follows: Among them, each q represents a question text, each x represents a label, the superscript t indicates that the question text and the label belong to the tth item in the training data set, the subscript is the serial number of the corresponding image, and T is the number of images; tokenizer represents the text feature extraction module, p t Represents the text features extracted from the t-th item of data in the training dataset.
5. A natural scene object detection method according to claim 1 or 2, characterized in that: The interleaving of visual features with text features through the text and visual feature interleaving layer includes: The symbols added to the text features to mark the image information will be used to represent the visual features f t p With the text feature p t Perform feature union and insert the visual features into the specified position of the text features, expressed as: Among them, F t cat Represents the data features after feature interleaving, represents the visual features of the i-th image in the t-th data, T represents the number of images contained in a data item, and p t,tag It represents the position where the image is marked in the text feature. The image feature is added to the corresponding position of the text feature through the above formula to complete the interleaving of visual and text features.
6. A natural scene object detection method according to claim 1, characterized in that: The object position information in the last image of each data item predicted by the language model includes: For the t-th item in the training data set, the language model prediction output is expressed as: Among them, T represents the number of images contained in a data item, i is the sequence number of the image, p(.|.) represents the conditional probability, and the generated result is the text data with the highest probability, r i Represents the data features after the features corresponding to the i-1th image are interleaved Generated text data H i-1 , label g t The text data generated under the condition; t represents the tth item of data in the training data set, is the label corresponding to the last image of the t-th item in the training dataset; express The predicted probability of are the position coordinates of the two focus points of the rectangular frame, and the rectangular frame is the object position information; according to Determine the object location information in the last image of the t-th item in the training dataset 7. A natural scene object detection method according to claim 1 or 6, characterized in that: The constructed loss function is expressed as: Among them, L d is the loss function, M means that the training data set contains M items of data, Represents the predicted object location information in the last image of the t-th item in the training dataset. It is the label corresponding to the last image of the t-th item in the training dataset.
8. A natural scene object detection system, characterized in that: include: A data acquisition and processing unit, used to acquire a data set of natural scene object position information, and reprocess and organize the data set according to a context learning strategy to obtain a training data set; wherein each item of data in the training data set includes multiple images, and a question text and a label corresponding to each image; A model building unit, used to build a natural scene object detection model based on the visual language model, including: a visual feature extraction module, a text feature extraction module, a text and visual feature cross layer and a language model; A model training unit, used for training the natural scene object detection model using a training data set, including: extracting visual features from images in each item of data through a visual feature extraction module, extracting text features from question texts and labels corresponding to the first to second to last images in each item of data, and the question text of the last image through a text feature extraction module; interleaving visual features with text features through a text and visual feature cross layer, and then predicting object position information in the last image of each item of data through a large language model; combining the predicted object position information in the last image of each item of data with the corresponding label, calculating a loss function, and training the natural scene object detection model; The object detection unit is used to input the image and the corresponding question text into the trained natural scene object detection model to obtain the object position information in the image.
9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Visual language cross-modal learning method for structural health diagnosis large model
CN117253112A
License plate detection method and device, storage medium and electronic equipment
CN117636326A
Aerial image multi-target rapid detection algorithm based on memory module interlacing model
CN117853951A
Downstream task model generation method, task execution method, and device
WO2024099144A1
Cited By
Target detection model training method and device based on dialogue guidance, and grabbing pose generation method and device based on dialogue guidance
CN121438019A
Target detection model training method based on dialogue guidance, and grabbing pose generation method and device
CN121438019B
Video surveillance system that detects hazardous situations
KR103000430B1
