A task-driven target detection method and device

Abstract

The present invention relates to the field of image processing technology, and discloses a task-driven target detection method and device, the method comprising: constructing a task-driven target detection data set, the data set comprising: text data, image data; training the task-driven target detection data set by presetting a task-driven target detection model, and completing the task-driven target detection according to a preset loss function when the loss calculation result meets the preset conditions. The method provided by the present invention improves the accuracy of target detection and reduces the phenomenon of missed detection and false detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a task-driven target detection method and device. Background Art

[0002] In recent years, with the continuous acceleration of urbanization, the number of motor vehicles has continued to increase, and many problems such as urban traffic congestion and parking conflicts have also arisen. Improving the level of intelligence in urban traffic management is a social problem that needs to be solved urgently. At present, the main method is to install high-position video surveillance equipment on the roadside of urban roads, important traffic light intersections, etc., collect video image data of the monitoring scene, and use visual algorithms to process and analyze the data to achieve analysis of the monitoring scene: such as capturing evidence and warning of traffic violations such as running a red light and speeding, guiding roadside parking and recording parking spaces, updating, predicting and publishing traffic congestion conditions in real time, detecting and warning abnormal behaviors, etc. Monitoring equipment is usually installed in relatively dense areas such as pedestrian and vehicle traffic, and the scene is relatively complex. At present, the main visual algorithms used are target detection, semantic segmentation, target tracking, etc. Among them, the two-dimensional target detection algorithm is one of the most widely used algorithms. For example, for roadside monitoring scenes, in an open outdoor environment, the scene is relatively complex and covers more target categories. Usually, a fixed target category table is set to label the target category data and train the model, and then detect the set target category when applying it.

[0003] However, with the development of current technology, and the increasingly in-depth and complex understanding and analysis of monitoring scenes, the task requirements are becoming more and more intelligent. Relying on traditional target detection algorithms, simply detecting the categories in the target detection category library can no longer meet the requirements of some tasks. Task-driven target detection algorithms refer to detecting object instances suitable for providing information for the task in the image to be detected. The challenge is that the target categories used for the task are diverse and not fixed, rather than limited to a set of fixed target category libraries in traditional target detection algorithms. It is necessary to select and locate targets based on the task based on a comprehensive understanding of specific tasks and image scenes; for example, if you want to detect whether there is illegal road occupation in the image, for traditional target detection algorithms, it is usually detected whether there are categories such as fruits, vegetables, and parasols in the image to determine whether there is such behavior. However, this method based on detecting fixed category targets is too rigid, and there are defects such as inaccurate target detection, false detection, or missed detection. Summary of the invention

[0004] In view of this, the present invention provides a task-driven target detection method and device to overcome the problem of inaccurate target detection, false detection or missed detection.

[0005] In a first aspect, the present invention provides a task-driven target detection method, the method comprising:

[0006] Constructing a task-driven target detection dataset, the dataset comprising: text data and image data;

[0007] The task-driven target detection data set is trained by presetting the task-driven target detection model. According to the preset loss function, when the loss calculation result meets the preset conditions, the task-driven target detection is completed.

[0008] In an optional implementation, the preset task-driven target detection model includes: an image feature extraction backbone network, a text feature encoding module, a large language model extraction visual cue knowledge module, a knowledge encoding and aggregation module, and an attribute knowledge decoding module, wherein:

[0009] Image feature extraction backbone network, used to generate image features of different scales;

[0010] Text feature encoding module, used to obtain text features of preset tasks;

[0011] The large language model extracts the visual cue knowledge module, which is used to obtain the target object that meets the preset task target detection;

[0012] The knowledge encoding and aggregation module is used to encode and fuse the target objects that meet the preset task target detection and generate image visual attribute knowledge;

[0013] The attribute knowledge decoding module is used to decode image features of different scales based on visual attribute knowledge to complete task-driven object detection.

[0014] In an optional implementation, the image feature extraction backbone network adopts a convolution combination of convolution layer-normalization layer-activation function layer, performs superposition operations of a preset number of convolution combination layers, and performs downsampling in each convolution combination operation.

[0015] In an optional implementation, the attribute knowledge decoding module adopts a transformer-based target detection structure.

[0016] In an optional implementation, the attribute knowledge decoding module includes: an encoder and a decoder, wherein:

[0017] The encoder is a transformer-based network structure;

[0018] The decoder uses random initialization to obtain the query of each target and completes task-driven target detection.

[0019] In an optional implementation, the preset loss function is calculated by the following formula:

[0020] L=α1L bbox +α2L cls

[0021] Among them, L bbox represents the position regression loss function and L cls Represents the classification loss function, α1 and α2 are constants.

[0022] In a second aspect, the present invention provides a task-driven target detection device, the device comprising:

[0023] A data construction module, used to construct a task-driven target detection data set, the data set including: text data, image data;

[0024] The task-driven target detection module is used to train the task-driven target detection dataset through a preset task-driven target detection model, and complete the task-driven target detection according to the preset loss function when the loss calculation result meets the preset conditions.

[0025] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the task-driven target detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the task-driven target detection method of the first aspect or any corresponding embodiment thereof.

[0027] The task-driven target detection method and device provided by the present invention construct a preset task-driven target detection model, learn the common sense knowledge of the open world from a large-scale corpus, and then extract the prompt knowledge of the driving task from the large model to guide the recognition and positioning of the target, thereby realizing a task-driven target detection algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 is a flowchart of a task-driven target detection method according to an embodiment of the present invention;

[0030] Figure 2 is a structural block diagram of a task-driven target detection device according to an embodiment of the present invention;

[0031] Figure 3 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0033] According to an embodiment of the present invention, an embodiment of a task-driven target detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] In this embodiment, a task-driven target detection method is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 1 is a flowchart of a task-driven target detection method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0035] Step S101, constructing a task-driven object detection data set, the data set including: text data and image data.

[0036] In an embodiment of the present invention, for task-driven target detection data, a task S and image data to be detected are given in the form of this article, and a task-driven target detection algorithm is used to output a target object that best meets the task. This is only an example and is not limited to this. In actual applications, the corresponding data set is selected according to actual conditions.

[0037] Specifically, for the construction of the data set, one is to use the task-based object detection data set in the open source data set, such as COCO data, etc. This is only an example and is not limited to this. In actual applications, the corresponding data is selected according to the actual situation. The second is to use video surveillance equipment to collect video image data, combine it with task requirements, and perform data annotation. The annotation method is not limited here, and it is selected according to the actual situation.

[0038] Step S102, training the task-driven target detection data set through a preset task-driven target detection model, and completing the task-driven target detection based on a preset loss function when the loss calculation result meets the preset conditions.

[0039] In an embodiment of the present invention, a preset task driven target detection model includes: an image feature extraction backbone network, a text feature encoding module, a large language model extraction visual cue knowledge module, a knowledge encoding and aggregation module, and an attribute knowledge decoding module. Among them, the image feature extraction backbone network is used to generate image features of different scales. The text feature encoding module is used to obtain text features of preset tasks. The large language model extraction visual cue knowledge module is used to obtain target objects that meet the preset task target detection. The knowledge encoding and aggregation module is used to encode and fuse the target objects that meet the preset task target detection to generate image visual attribute knowledge. The attribute knowledge decoding module is used to decode image features of different scales according to visual attribute knowledge to complete task-driven target detection.

[0040] In the embodiment of the present invention, the image feature extraction backbone network is used to extract image features and generate features of different scales. The input image size is H*W*C, where H and W represent the height and width of the image respectively, and C represents the number of channels. In this case, C is 3, indicating an RGB three-channel image. After the backbone network, feature maps of different scales are output, and the feature maps are expanded along the width and height dimensions of the image, which is defined as V, indicating feature outputs of different scales.

[0041] Specifically, the backbone network mainly adopts the convolution combination of convolution layer-normalization layer-activation function layer, performs superposition operation of multiple convolution combination layers, and performs downsampling once in each convolution combination operation, including but not limited to using backbone networks such as ResNet, VGG, MobileNet, etc. This is only an example, not limited to this. In actual applications, the corresponding backbone network is selected according to the actual situation, and the downsampling multiple is R, including N downsampling operations in total. Normalization layer, including but not limited to instance normalization layer, adaptive instance normalization layer, etc. Non-linear activation layer includes but is not limited to non-linear activation functions such as ReLU and Leaky ReLU.

[0042] In the embodiment of the present invention, the function of the text feature encoding module is to obtain the text feature t of task S. s , which is used to query the visual content related to the task in the image. The text encoder uses a pre-trained text encoding model, including but not limited to BERT, XLNet and other models. This is only an example and is not limited to this. In actual applications, the corresponding text editor is selected according to the actual situation.

[0043] In an embodiment of the present invention, the large language model extracts visual cue knowledge module for target detection based on a specific task, and locates the target object that meets the task requirement according to the requirement. However, the task requirement is usually abstract and cannot directly correspond to the visual content in the image to locate the target detection object. Therefore, by constructing a large language model to extract visual cue knowledge module, visual attribute knowledge is extracted, and the correspondence between the task requirement and the target instance of the corresponding task in the image is achieved based on the acquired visual attribute knowledge. In addition, the large language model is used to generate visual attribute knowledge because the large language model usually contains world-wide knowledge learned from a large amount of text data.

[0044] Specifically, in order to fully utilize the capabilities of the large language model and obtain objectives that better meet task requirements, the process of using the large language model to generate video attribute knowledge is divided into the following three steps:

[0045] (1) Using the large language model, input the following prompt information: "What common items in life can be used to complete task S? Please list m most suitable items." m can be set to 10 or 20, etc. The output of the large language model is defined as N o ;

[0046] (2) Using the large language model, we ask the user to state the rationality of the above output targets to complete the task, and for each target, we obtain a set of reasons why the task can be completed;

[0047] (3) Using a large language model, summarize the reasonable visual features of each target;

[0048] Finally, we get N o Groups of visual attribute knowledge, where each group contains multiple visual attributes that can complete the task.

[0049] In the embodiment of the present invention, the knowledge encoding and aggregation module is to obtain the N o Encode and fuse group visual attribute knowledge.

[0050] Specifically, for the above N oEach knowledge unit in the group visual attribute knowledge includes multiple text descriptions. The multiple text descriptions are concatenated to obtain a text sequence, and then a text encoder is used to encode the text sequence features to obtain text sequence features. The text encoder includes but is not limited to BERT, XLNet and other text encoders. This is only an example and is not limited to this. In actual applications, the corresponding text encoder is selected according to the actual situation.

[0051] Specifically, in order to exclude inappropriate knowledge units, the cosine similarity between any two knowledge units is calculated, and if the cosine similarity value is lower than a preset threshold, the knowledge unit is removed. This is only an example, and is not limited to this. In actual applications, the corresponding exclusion method is selected according to the actual situation.

[0052] Specifically, the text encoder is used to extract the text features of the target word in each visual knowledge unit, and finally the above features are fused to obtain visual attribute knowledge, which is specifically expressed as:

[0053]

[0054] Among them, K represents visual attribute knowledge, and They represent text sequence knowledge and word knowledge respectively, and N means there are N knowledge units in the end.

[0055] In the embodiment of the present invention, the decoding module of attribute knowledge adopts a transformer-based target detection structure, namely DETR, which includes two parts: an encoder and a decoder.

[0056] Specifically, the input of the encoder is the feature map V obtained in the above steps. The encoder is a transformer-based network structure that outputs further refined visual features, defined as F = {f1, f2, ..., f i}, indicating features of different scales.

[0057] Specifically, the decoder usually uses random initialization to obtain the query of each target, defined as Q = {q1, q2, ..., q k}, then query q from each target k Predict a reference point p k , these reference points are defined as L = {p1, p2, ..., p k} as the center point of the target box. Finally, the final target detection result O is obtained based on the query Q and the reference point L, which is defined as:

[0058] O = TransDecoder([Q, L], F)

[0059] Among them, TransDecoder represents the transformer-based decoder, and F represents the above-mentioned visual features.

[0060] Specifically, in order to make full use of visual attribute knowledge, the query Q and reference points are generated by combining visual image information, task requirements, and visual attribute knowledge instead of using random initialization, which is defined as Q kn , the specific calculation method is as follows:

[0061]

[0062] Among them, f i and t s They represent the visual and text features in the above steps respectively, fc() represents the fully connected layer, cos() represents the calculation of cosine similarity, and s i,j Represents the similarity between the i-th visual feature and the j-th knowledge unit;

[0063] r i =max(s i,j ),d i = argmax(s i,j )

[0064] Specifically, r i represents the relevance score of the i-th visual feature, d i Represents the index of the knowledge unit;

[0065]

[0066] Specifically, topk represents the selection of k relevant features with the highest correlation scores; AttentionPool() represents the attention layer and pooling layer operations;

[0067] Specifically, the Q obtained according to the above steps is kn , perform the final decoding operation, expressed as:

[0068] O kn =TransDecoder([Q kn , L kn ],F)

[0069] Among them, TransDecoder represents a transformer-based decoder.

[0070] In a specific embodiment, the loss function is a target detection loss function, including a position regression loss function L bbox And the classification loss function L cls Two parts.

[0071] Specifically, the loss function is calculated by the following formula:

[0072] L=α1L bbox +α2L cls

[0073] Among them, α1 and α2 are weight coefficients, which can be set to 1. The set values ​​are not limited here and are selected according to the actual situation. For the position regression loss function, including but not limited to the use of L1Loss mean absolute error loss function, CIOU Loss loss function, GIoU Loss loss function and other position regression loss functions, there is no restriction here. For the classification loss function, including but not limited to the use of Cross Entropy cross entropy loss function, Focal Loss loss function and other classification loss functions, there is no restriction here. When the loss calculation result meets the preset conditions, the task-driven target detection is completed. The preset conditions are selected according to the actual situation and are not limited here. For example, when the loss calculation result is less than the preset value, the preset value is not limited here and is selected according to the actual situation.

[0074] The task-driven target detection method provided in this embodiment builds a preset task-driven target detection model, learns common sense knowledge of the open world from a large-scale corpus, and then extracts prompt knowledge of the driving task from the large model to guide the recognition and positioning of the target, thereby realizing a task-driven target detection algorithm.

[0075] In the present embodiment, a task-driven target detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" may implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0076] This embodiment provides a task-driven target detection device, such as Figure 2 As shown, including:

[0077] The data construction module 201 is used to construct a task-driven target detection data set, and the data set includes: text data and image data.

[0078] The task-driven target detection module 202 is used to train the task-driven target detection data set through a preset task-driven target detection model, and complete the task-driven target detection according to a preset loss function when the loss calculation result meets the preset conditions.

[0079] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0080] The task-driven target detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0081] The embodiment of the present invention also provides a computer device having the above Figure 3 The task-driven target detection device shown.

[0082] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 3 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 10 is taken as an example.

[0083] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0084] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0085] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0086] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0087] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0088] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0089] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A task-driven target detection method, characterized in that: The method comprises: Constructing a task-driven target detection dataset, the dataset comprising: text data and image data; The task-driven target detection data set is trained by presetting the task-driven target detection model. According to the preset loss function, when the loss calculation result meets the preset conditions, the task-driven target detection is completed.

2. The method according to claim 1, characterized in that The preset task-driven target detection model includes: image feature extraction backbone network, text feature encoding module, large language model extraction visual cue knowledge module, knowledge encoding and aggregation module, attribute knowledge decoding module, among which, Image feature extraction backbone network, used to generate image features of different scales; Text feature encoding module, used to obtain text features of preset tasks; The large language model extracts the visual cue knowledge module, which is used to obtain the target object that meets the preset task target detection; The knowledge encoding and aggregation module is used to encode and fuse the target objects that meet the preset task target detection and generate image visual attribute knowledge; The attribute knowledge decoding module is used to decode image features of different scales based on visual attribute knowledge to complete task-driven object detection.

3. The method according to claim 2, characterized in that The image feature extraction backbone network adopts the convolution combination of convolution layer-normalization layer-activation function layer, performs superposition operation of a preset number of convolution combination layers, and performs downsampling in each convolution combination operation.

4. The method according to claim 2, characterized in that: The attribute knowledge decoding module adopts a transformer-based object detection structure.

5. The method according to claim 4, characterized in that The decoding module of attribute knowledge includes: encoder and decoder, where: The encoder is a transformer-based network structure; The decoder uses random initialization to obtain the query of each target and completes task-driven target detection.

6. The method according to claim 2, characterized in that The default loss function is calculated using the following formula: L=α1L bbox +α2L cls Among them, L bbox represents the position regression loss function and L cls Represents the classification loss function, α1 and α2 are constants.

7. A task-driven target detection device, characterized in that: The device comprises: A data construction module, used to construct a task-driven target detection data set, the data set including: text data, image data; The task-driven target detection module is used to train the task-driven target detection dataset through a preset task-driven target detection model, and complete the task-driven target detection according to the preset loss function when the loss calculation result meets the preset conditions.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the task-driven target detection method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the task-driven target detection method according to any one of claims 1 to 6.

Images