Target detection algorithm and system based on depth relation characteristics, processing equipment and storage medium

Abstract

The invention relates to a target detection algorithm and system based on depth relation characteristics, a processing device and a storage medium, and the method comprises the steps: marking image data, and adding a target relation label; adopting an inverted pyramid backbone network to extract target annotation relation information and target features of the annotated image data; extracting background features, position relation features and depth relation features of the image data according to the target annotation relation information and target features of the annotated image data by adopting a depth relation feature extraction method; obtaining a final fusion feature by adopting a feature fusion network according to the background feature, the position relation feature and the depth relation feature of the image data and the extracted target feature; a regression network and a classification network are adopted, and coordinates and category information of the target are determined according to the final fusion features; the target coordinate and category information is restored to obtain a final target detection result, and the target detection method can be widely applied to the field of target detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a target detection algorithm, system, processing device and storage medium based on deep relationship features. Background Art

[0002] At present, the target detection algorithms in the existing technology are still focused on identifying targets individually, and do not attempt to explore the relationships between targets during training. Limited by the current simple network structure, they are unable to model irregular target relationships.

[0003] Therefore, referring to the attention mechanism in natural language processing, the prior art proposes an adaptive attention mechanism for target detection. Its detection targets have 2D spatial distribution and different aspect ratios, which are more complicated than text scenarios. Therefore, the attention weights of NLP are expanded and divided into two weights: weights based on target features, which are similar to NLP weights; and weights based on the relative geometric position of the target. The relative geometric position ensures translation invariance. The attention mechanism includes an object relation module, an object recognition part, and a duplicate target removal part. The object relation module is used to learn the relationship between targets to enhance features and remove duplicate results. It receives variable inputs and performs parallel calculations. It is a differentiable in-place (a special operation mode that allows data to be modified directly in place instead of creating new data objects) operation. It can be embedded as a basic building block into any target detection algorithm. The embedding method is as follows: Figure 1 As shown, it is used for target recognition and duplicate target removal; the target recognition part is used to use the target relationship module to combine all the detected targets to perform feature enhancement and then perform subsequent recognition; duplicate target removal is not distinguished, and is used to replace this practice with a lightweight relationship network after the recognition is completed.

[0004] However, the adaptive attention mechanism for target detection proposed in the prior art still has difficulty in capturing the logical relationship between targets, and there is still a certain amount of false detection of targets. Summary of the invention

[0005] In response to the above problems, the purpose of the present invention is to provide a target detection algorithm, system, processing device and storage medium based on deep relationship features, which can extract more complex logical relationships and improve the accuracy of target detection.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions: In a first aspect, a target detection algorithm based on deep relationship features is provided, comprising: Label the image data and add target relationship labels; An inverted pyramid backbone network is used to extract target annotation relationship information and target features from the labeled image data; A depth relationship feature extraction method is used to extract background features, position relationship features and depth relationship features of the image data according to target annotation relationship information and target features of the annotated image data; Using the feature fusion network, the final fusion feature is obtained according to the background features, position relationship features, depth relationship features and extracted target features of the image data; Using regression network and classification network, the coordinates and category information of the target are determined according to the final fusion features; The coordinates and category information of the target are restored to obtain the final target detection result.

[0007] Furthermore, the target relationship tags include position relationship tags, depth relationship tags and reference system relationship tags.

[0008] Furthermore, the depth relationship feature extraction method is used to extract background features, position relationship features and depth relationship features of the image data according to the target annotation relationship information and target features of the annotated image data, including: A spatiotemporal-aware cascade neural network is used to extract and fuse the target annotation relationship information and target features of the annotated image data, and to extract the background features of the image data. The radial basis neural network is used to extract the positional relationship features of image data; A fully connected neural network is used to obtain the deep relationship features of the image data based on the background features and position relationship features of the image data.

[0009] Furthermore, the spatiotemporal perception cascade neural network is used to extract and fuse the target annotation relationship information and target features of the annotated image data to extract the background features of the image data, including: Extracting the target annotation relationship information and target features of the annotated image data to obtain temporal features and spatial features; The concat operation is used to connect the extracted temporal features and spatial features, and a fully connected neural network is used to fuse them to obtain the background features of the image data.

[0010] Furthermore, the use of radial basis function neural network to extract positional relationship features of image data includes: According to the radial basis function in the radial basis network, the radial distance between each feature in the extracted target features is calculated; A fully connected neural network is used to perform radial distance fusion judgment between features and extract the positional relationship features of image data.

[0011] Furthermore, the feature fusion network is used to obtain the final fusion features according to the background features, position relationship features, depth relationship features and extracted target features of the image data, including: The background feature a, the position relationship feature b, the depth relationship feature c and the target feature d of the image data are spliced ​​to obtain a spliced ​​first feature a+b+d and a second feature b+c+d; A feature fusion network is used to extract the concatenated first feature a+b+d and the second feature b+c+d to obtain the final fusion feature.

[0012] Furthermore, the post-processing is used to restore the coordinates and category information of the target to obtain the final target detection result, including: Restore the coordinates of the target according to the width and height of the input image data; Take the index of the maximum value of the target category information to get the final category information.

[0013] In a second aspect, a target detection system based on deep relationship features is provided, comprising: Annotation module, used to annotate image data and add target relationship labels; A first feature extraction module is used to extract target annotation relationship information and target features of the labeled image data using an inverted pyramid backbone network; The second feature extraction module is used to extract background features, position relationship features and depth relationship features of the image data according to the target annotation relationship information and target features of the annotated image data by using a depth relationship feature extraction method; A feature fusion module is used to obtain the final fusion feature according to the background feature, position relationship feature, depth relationship feature and extracted target feature of the image data by using a feature fusion network; The regression and classification module is used to use the regression network and the classification network to determine the coordinates and category information of the target according to the final fusion features; The restoration module is used to restore the coordinates and category information of the target to obtain the final target detection result.

[0014] According to a third aspect, a processing device is provided, comprising computer program instructions, wherein the computer program instructions, when executed by the processing device, are used to implement the steps corresponding to the above-mentioned target detection method based on deep relationship features.

[0015] In a fourth aspect, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, are used to implement the steps corresponding to the above-mentioned target detection method based on deep relationship features.

[0016] The present invention adopts the above technical solution, which has the following advantages: 1. By increasing the data annotation format and redesigning the relationship feature extraction module, the present invention has the ability to extract more complex logical relationships, which is not limited to target deduplication. It can improve the accuracy of target detection, solve the problem that the target detection model is difficult to capture the logical relationship between targets, and greatly reduce the false detection of targets.

[0017] 2. Compared with the relationship module target detection model method used in the prior art, the present invention not only removes duplicate targets, but also reduces the false alarm rate, and can improve the accuracy and robustness of the target detection algorithm.

[0018] 3. The present invention modifies the post-processing logic into a data annotation method, so that a guided training model can be achieved.

[0019] In summary, the present invention can be widely used in the field of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings: Figure 1 It is a flow chart of an adaptive attention mechanism for target detection in the prior art provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a method flow provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0022] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0023] Although the terms first, second, third, etc. can be used in the text to describe multiple elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or section from another region, layer or section. Unless the context clearly indicates, terms such as "first", "second" and other numerical terms do not imply order or sequence when used in the text. Therefore, the first element, component, region, layer or section discussed below can be referred to as the second element, component, region, layer or section without departing from the teaching of the example embodiments.

[0024] At present, the adaptive attention mechanism for target detection proposed in the prior art is still difficult to capture the logical relationship between targets, and there are still certain false detections of targets. Therefore, this embodiment provides a target detection algorithm based on deep relationship features, including: annotating image data and adding target relationship labels; using an inverted pyramid backbone network to extract target annotation relationship information and target features of the annotated image data; using a deep relationship feature extraction method to extract background features, position relationship features and depth relationship features of the image data according to the target annotation relationship information and target features of the annotated image data; using a feature fusion network to obtain the final fusion feature according to the background features, position relationship features, depth relationship features and extracted target features of the image data; using a regression network and a classification network to determine the coordinates and category information of the target according to the final fusion feature; restoring the coordinates and category information of the target to obtain the final target detection result. The present invention has the ability to extract more complex logical relationships by increasing the data annotation format and redesigning the relationship feature extraction module, which is not limited to target deduplication, improves the accuracy of target detection, and can solve the problem that the target detection model is difficult to capture the logical relationship between targets, greatly reducing the false detection of targets.

[0025] Example 1 like Figure 2 As shown, this embodiment provides a target detection algorithm based on deep relationship features, including the following steps: 1) Label the image data.

[0026] Specifically, when annotating image data, target relationship labels are added.

[0027] Specifically, the target relationship labels include position relationship labels, depth relationship labels, and reference frame relationship labels. For example, the position relationship labels are containment, separation, and intersection; the depth relationship labels are the category and size of the nearest neighbor target; and the reference frame relationship labels are the position and size of the reference frame specified in different environments.

[0028] 2) An inverted pyramid backbone network is used to extract target annotation relationship information and target features (including position, depth, and reference frame) from the annotated image data.

[0029] Specifically, the inverted pyramid backbone network refers to detecting objects of different sizes through feature maps of different resolutions in target detection. This structure usually includes a series of convolutional layers, each convolutional layer consists of a series of convolutional blocks, and the size of the feature map decreases with the depth of the network.

[0030] 3) Using the deep relationship feature extraction method, the background features, position relationship features and deep relationship features of the image data are extracted according to the target annotation relationship information and target features of the annotated image data, specifically: 3.1) Using a spatiotemporal perception cascade neural network, we extract and fuse the target annotation relationship information and target features of the annotated image data to extract the background features of the image data: 3.1.1) Extract the target annotation relationship information and target features of the annotated image data to obtain temporal features and spatial features.

[0031] Specifically, feature extraction is divided into temporal feature extraction and spatial feature extraction. Temporal feature extraction is to perform deep convolution operations between channels of two consecutive frames of different images (similar to the frame difference method); spatial feature is to perform deep convolution operations on a single frame image.

[0032] 3.1.2) Use the concat operation to connect the extracted temporal features and spatial features, and use a fully connected neural network to fuse them to obtain the background features of the image data.

[0033] 3.2) Use radial basis neural network to extract the positional relationship features of image data: 3.2.1) According to the radial basis function in the radial basis network, the radial distance between each feature in the extracted target features is calculated.

[0034] 3.2.2) A fully connected neural network is used to perform radial distance fusion judgment between features and extract the positional relationship features of the image data (according to the pre-defined positional relationship features, it will eventually converge to the positional relationship).

[0035] 3.3) Using a fully connected neural network, we can obtain the deep relationship features of the image data based on the background features and position relationship features of the image data: 3.3.1) Using the concat operation, the background features of the image data obtained in step 3.1) and the positional relationship features of the image data obtained in step 3.2) are connected.

[0036] 3.3.2) A fully connected neural network is used to fuse the connected features to obtain the deep relationship features of the image data (since the data is labeled with deep relationship feature labels, a staged output head is used here, and finally the training can converge to the deep relationship features). The present invention makes extensive use of staged connections and staged loss functions, wherein the staged connections are steps 3.1), 3.2) and 3.3) The outputs will all be part of the final output, and the staged loss function is steps 3.1), 3.2) and 3.3) The loss function in order to ensure the purity of the features will only affect the training of the current stage and will not affect the training of other stages.

[0037] 4) Using the feature fusion network, the final fusion features are obtained based on the background features, position relationship features, depth relationship features and extracted target features of the image data, specifically: 4.1) The background feature a, position relationship feature b, depth relationship feature c and target feature d of the image data are simply spliced ​​to obtain the spliced ​​first feature a+b+d and the second feature b+c+d.

[0038] 4.2) Use the feature fusion network to extract the concatenated first feature a+b+d and the second feature b+c+d to obtain the final fusion feature.

[0039] Specifically, the feature fusion network uses multiple layers (generally no more than 3 layers) of fully connected layers. Due to the cross-connection and nonlinear transformation functions of the fully connected network, the features can be operated on each other and finally nonlinearly mapped to obtain fused features.

[0040] 5) Use regression network and classification network to determine the coordinates and category information of the target based on the final fusion features.

[0041] Specifically, the basic feature extraction and fusion network parts of the regression network and classification network all use convolution + activation function and concat method. The main difference is that the loss function of the regression network is generally the mean square error (MSE), and the loss function of the classification network is generally softmax.

[0042] 6) Post-processing is used to restore the coordinates and category information of the target to obtain the final target detection results, specifically: 6.1) Restore the coordinates of the target based on the width and height of the input image data.

[0043] Specifically, the coordinates of the target determined in step 5) are normalized values ​​greater than 0 and less than 1, and need to be calculated based on the width and height of the input image data. For example: suppose the coordinates of the determined target are (x, y), the width of the image data is W and the height is H, then the values ​​of the coordinates (X, Y) of the restored target are: X=W*x Y=H*y 6.2) Take the index of the maximum value of the target category information to obtain the final category information.

[0044] Specifically, the category information of the target determined in step 5) is a plurality of decimals, each digit represents the probability of the index category, and the index where the maximum value is located needs to be taken, and the final category information is obtained through the index of the maximum value according to the predefined category list in the data set.

[0045] For example, if the predefined category list is [people, dogs, cars], and the category information of the target determined in step 5) is [0.5, 0.6, 0.9], then take the maximum value 0.9 at index 2, and use the 2-index category list to get the final target category as "car".

[0046] Example 2 This embodiment provides a target detection system based on deep relationship features, which is characterized by comprising: Annotation module, used to annotate image data and add target relationship labels; A first feature extraction module is used to extract target annotation relationship information and target features of the labeled image data using an inverted pyramid backbone network; The second feature extraction module is used to extract background features, position relationship features and depth relationship features of the image data according to the target annotation relationship information and target features of the annotated image data by using a depth relationship feature extraction method; A feature fusion module is used to obtain the final fusion feature according to the background feature, position relationship feature, depth relationship feature and extracted target feature of the image data by using a feature fusion network; The regression and classification module is used to use the regression network and the classification network to determine the coordinates and category information of the target according to the final fusion features; The restoration module is used to restore the coordinates and category information of the target to obtain the final target detection result.

[0047] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0048] Example 3 This embodiment provides a processing device corresponding to the target detection algorithm based on deep relationship features provided in Embodiment 1. The processing device can be applicable to a client processing device, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of Embodiment 1.

[0049] The processing device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete mutual communication. The memory stores a computer program that can be run on the processing device, and when the processing device runs the computer program, the target detection algorithm based on deep relationship features provided in this embodiment 1 is executed.

[0050] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage.

[0051] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0052] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0053] Those skilled in the art will understand that the structure of the above-mentioned computing device is only a partial structure related to the solution of the present invention, and does not constitute a limitation on the computing device to which the solution of the present invention is applied. The specific computing device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0054] Example 4 This embodiment provides a computer program product corresponding to the target detection algorithm based on deep relationship features provided in Embodiment 1. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the target detection algorithm based on deep relationship features described in Embodiment 1 are loaded.

[0055] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0056] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0057] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0060] The above embodiments are only used to illustrate the present invention, wherein the structure, connection mode and manufacturing process of each component may be changed. Any equivalent transformations and improvements based on the technical solution of the present invention should not be excluded from the protection scope of the present invention.

Claims

1. A target detection algorithm based on deep relationship features, characterized in that: include: Label the image data and add target relationship labels; An inverted pyramid backbone network is used to extract target annotation relationship information and target features from the labeled image data; A depth relationship feature extraction method is used to extract background features, position relationship features and depth relationship features of the image data according to target annotation relationship information and target features of the annotated image data; Using the feature fusion network, the final fusion feature is obtained according to the background features, position relationship features, depth relationship features and extracted target features of the image data; Using regression network and classification network, the coordinates and category information of the target are determined according to the final fusion features; The coordinates and category information of the target are restored to obtain the final target detection result.

2. The target detection algorithm based on deep relationship features as claimed in claim 1, characterized in that: The target relationship tags include position relationship tags, depth relationship tags and reference system relationship tags.

3. The target detection algorithm based on deep relationship features as claimed in claim 1, characterized in that: The method for extracting deep relationship features is used to extract background features, position relationship features and deep relationship features of image data according to target annotation relationship information and target features of the annotated image data, including: A spatiotemporal-aware cascade neural network is used to extract and fuse the target annotation relationship information and target features of the annotated image data, and to extract the background features of the image data. The radial basis neural network is used to extract the positional relationship features of image data; A fully connected neural network is used to obtain the deep relationship features of the image data based on the background features and position relationship features of the image data.

4. The target detection algorithm based on deep relationship features as claimed in claim 3, characterized in that: The method uses a spatiotemporal perception cascade neural network to extract and fuse the target annotation relationship information and target features of the annotated image data to extract background features of the image data, including: Extracting the target annotation relationship information and target features of the annotated image data to obtain temporal features and spatial features; The concat operation is used to connect the extracted temporal features and spatial features, and a fully connected neural network is used to fuse them to obtain the background features of the image data.

5. The target detection algorithm based on deep relationship features as claimed in claim 3, characterized in that: The radial basis function neural network is used to extract the positional relationship features of the image data, including: According to the radial basis function in the radial basis network, the radial distance between each feature in the extracted target features is calculated; A fully connected neural network is used to perform radial distance fusion judgment between features and extract the positional relationship features of image data.

6. The target detection algorithm based on deep relationship features as claimed in claim 1, characterized in that: The feature fusion network is used to obtain the final fusion features according to the background features, position relationship features, depth relationship features and extracted target features of the image data, including: The background feature a, the position relationship feature b, the depth relationship feature c and the target feature d of the image data are spliced ​​to obtain a spliced ​​first feature a+b+d and a second feature b+c+d; A feature fusion network is used to extract the concatenated first feature a+b+d and the second feature b+c+d to obtain the final fusion feature.

7. The object detection algorithm based on deep relationship features as claimed in claim 1, characterized in that: The post-processing is used to restore the coordinates and category information of the target to obtain the final target detection result, including: Restore the coordinates of the target according to the width and height of the input image data; Take the index of the maximum value of the target category information to get the final category information.

8. A target detection system based on deep relationship features, characterized in that: include: Annotation module, used to annotate image data and add target relationship labels; A first feature extraction module is used to extract target annotation relationship information and target features of the labeled image data using an inverted pyramid backbone network; The second feature extraction module is used to extract background features, position relationship features and depth relationship features of the image data according to the target annotation relationship information and target features of the annotated image data by using a depth relationship feature extraction method; A feature fusion module is used to obtain the final fusion feature according to the background feature, position relationship feature, depth relationship feature and extracted target feature of the image data by using a feature fusion network; The regression and classification module is used to use the regression network and the classification network to determine the coordinates and category information of the target according to the final fusion features; The restoration module is used to restore the coordinates and category information of the target to obtain the final target detection result.

9. A processing device, characterized in that: It includes computer program instructions, wherein when the computer program instructions are executed by a processing device, they are used to implement the steps corresponding to the target detection method based on deep relationship features described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, wherein the computer program instructions, when executed by a processor, are used to implement steps corresponding to the object detection method based on deep relationship features described in any one of claims 1 to 7.

Images