A small target detection method and device based on a knowledge graph and a storage medium

Abstract

The application provides a small target detection method and device based on a knowledge graph and a storage medium. Low-resolution small targets have little visual information, and it is difficult to extract features with discriminative power, and they are easily disturbed by environmental factors, which makes it difficult for a detection model to accurately locate and identify small targets. The application constructs a target context knowledge graph based on a CN-Probase Chinese general concept knowledge graph, and improves a YOLOv3 network so that the target detection model can fully utilize prior information in the knowledge graph and obtain more context features, thereby effectively improving the small target detection precision while ensuring the detection speed.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method, device and storage medium for small target detection based on knowledge graph. Background Technology

[0002] In object detection algorithms, small targets suffer from low resolution compared to large / medium-sized targets. Low-resolution small targets offer limited visual information, making it difficult to extract discriminative features. Furthermore, they are highly susceptible to environmental interference, hindering the accurate localization and identification of small targets by the detection model. Currently, several small target detection algorithms exist:

[0003] 1. Small target detection algorithm based on multi-scale learning

[0004] Compared to regular targets, small targets have fewer usable pixels, making it difficult to extract good features. Furthermore, as the number of network layers increases, the feature and location information of small targets is gradually lost, making them difficult for the network to detect. Multi-scale feature fusion considers both shallow representational information and deep semantic information, which is beneficial for feature extraction of small targets and can effectively improve the detection performance of small targets. Fast-Paced Networks (FPNs) are currently the most popular multi-scale networks. They introduce a bottom-up, top-down network structure, fusing features from adjacent layers to achieve feature enhancement. However, multi-scale learning methods also increase computational cost, and the influence of interference noise is difficult to avoid during feature fusion.

[0005] 2. Small object detection algorithm based on context learning

[0006] Appropriately modeling the context can improve object detection performance, especially for small targets with inconspicuous appearance features. Context-based object detection utilizes background features around the target region or global scene features, making full use of target-related information in the image, which can effectively improve the performance of small target detection. However, current research methods do not consider the problem of potentially scarce contextual information in the scene.

[0007] 3. Small object detection algorithm based on generative adversarial learning

[0008] Generative adversarial learning (GAN) aims to achieve detection performance comparable to that of larger targets by mapping the features of low-resolution small targets to features equivalent to those of high-resolution targets. By incorporating GANs, the resolution of small targets can be improved, the feature differences between small and large / medium-scale targets can be reduced, and the feature representation of small targets can be enhanced, thereby improving the performance of small target detection. However, GAN models are difficult to train, struggle to achieve a good balance between the generator and discriminator, and their performance improvement is limited after training reaches a certain level. Summary of the Invention

[0009] To address the above problems, the first aspect of this invention provides a small target detection method based on knowledge graphs, comprising the following steps:

[0010] S1, Construct a target context knowledge graph based on CN-Probase Chinese general concept knowledge graph and ImageNet image dataset;

[0011] S2, in the target context knowledge graph, find related entities based on the category information and text description of the small targets in the training set to obtain explicit context information to assist in detection;

[0012] S3, Based on the YOLOv3 network model, an improvement is made, and a small target detection model is built using the explicit context information obtained in step S2;

[0013] S4. Input the small target image into the small target detection model built in step S3, perform detection processing on the image, and output the detection results.

[0014] In one possible design, the specific process of constructing the target context knowledge graph based on the CN-Probase Chinese general concept knowledge graph and the ImageNet image dataset in step S1 is as follows:

[0015] S11. Named entity recognition is performed on the category information of the ImageNet dataset in text format using the BiLSTM+CRF algorithm. First, the Skip-gram model is used to map words to word vectors. Then, the BiLSTM layer is used to obtain the score probability of each word for each label. Finally, the score probability is input into the CRF layer to learn the order dependency information between labels and obtain the named entity recognition result.

[0016] S12, link the structured data of CN-Probase Chinese General Concept Knowledge Graph with the entities extracted in step S11: select a set of candidate entity objects from the knowledge base according to the given entity references, and then link the references to the correct entity objects through Cosine similarity calculation;

[0017] S13. Construct an ontology based on the images in the ImageNet dataset and the factual representations obtained in step S12. Use the RDFS language to perform schema layer modeling. For entities with part-of / constraint / dependency relationships, select images in the ImageNet dataset where the entities appear simultaneously, and perform target annotation. Calculate the relative values ​​of the length and width of each target to the length and width of the image, and fill the corresponding entity's attribute value range with these values.

[0018] S14: Perform a quality assessment on the formed target context knowledge graph, and expand the knowledge graph through knowledge reasoning.

[0019] In one possible design, step S2 specifically involves using SPARQL to query entities with which the target context knowledge graph has part-of / constraint / dependency relationships for small target categories that are easily missed in the training set, and calculating the ratio of the width and height attributes between entities, thereby obtaining explicit contextual information to assist in detection.

[0020] In one possible design, step S3, which utilizes explicit contextual information to build a small target detection model, involves the following steps:

[0021] S31, the original dataset is divided into training set, validation set and test set in an 8:1:1 ratio;

[0022] S32, set the initial training parameters, including: using the DarkNet53 model pre-trained on the ImageNet dataset as the backbone; resizing the image size to 640*640 pixels; epochs to 300; batch size to 16; using the Adam optimizer with an initial learning rate of 0.0001, beta1 of 0.937, beta2 of 0.999, and weight decay of 0.0005; warmup epochs to 3.0, initial warmup momentum of 0.8, and initial warmup bias of 0.1.

[0023] S33: The training set is input into the initial weight model in batches for forward propagation to obtain the predicted classification and coordinate regression results. Specifically, the Mosaic method is used for data augmentation, which involves randomly using several images, randomly scaling and cropping them, and then randomly distributing and stitching them together. The Mosaic data augmentation method is turned off in the last 15 batches of training. At the same time, based on the assumption that "regions of the same scale and aspect ratio of different small targets have similar contextual features", the location label and category label of the large target are generated proportionally according to the obtained aspect ratio data of the small and large targets.

[0024] S34, calculate the loss and perform backpropagation to update the model weights. The training ends when iterates to 300 epochs and the model is saved every 30 epochs. Specifically, the classification loss is calculated by multi-class cross-entropy, the confidence loss is calculated by binary cross-entropy, and the coordinate loss is calculated by CIoU Loss for the labels and prediction results of the original small target and the generated large target, and then the weighted sum is used as the overall loss.

[0025] S35, test and obtain a converged small target detection model.

[0026] In one possible design, the improved YOLOv3 network structure includes an improved network comprising a backbone network module, a feature pyramid module, an anchor matching module, and a decoupled head prediction module;

[0027] The backbone network module uses the DarkNet53 network with the fully connected layers removed for feature extraction, and uses the feature maps of the last three layers for prediction.

[0028] The feature pyramid module upsamples the small feature map with a step size of 2, and then concatenates it with the large feature map to fuse features;

[0029] The Anchor matching module first generates a feature map of the same size through the feature enhancement unit. Then, based on the obtained aspect ratio data of the small and large targets, it expands the original anchor size with the original small target position as the center point to generate a context anchor and assigns the context anchor to a feature layer with a larger receptive field. The feature enhancement unit performs 1*1 convolution and upsampling on the upper feature map and concatenates it with the current feature map. Then, it uses a dilated convolution with a 3*3 kernel and a dilation rate of 3 to capture multi-scale representations and enhance the receptive field.

[0030] The decoupled head prediction module first uses 1*1 convolutions for dimensionality reduction, then obtains the classification prediction result of the layer through a classification branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function. It then obtains the confidence prediction result of the layer through a confidence branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function. Finally, it obtains the coordinate prediction result of the layer through a coordinate branch containing two 3*3 convolutions and one 1*1 convolution. Finally, it concatenates and transposes the results of all layers to obtain the final output.

[0031] A second aspect of the present invention provides a small target detection device based on a knowledge graph, the device comprising at least one processor and at least one memory coupled together; the memory stores a computer program for a small target detection model constructed as described in the first aspect; when the processor executes the computer program stored in the memory, the device performs small target detection.

[0032] A third aspect of the present invention provides a computer-readable storage medium storing a program or instructions for constructing a small target detection model as described in the first aspect, wherein when the program or instructions are executed by a processor, the computer performs small target detection.

[0033] Beneficial effects: Compared with the prior art, the present invention provides a small target detection method, device and storage medium based on knowledge graph, which greatly improves the effect of small target detection and produces the following beneficial effects:

[0034] 1. Small targets have low resolution and limited visualization information, making it difficult to extract effective features. This invention establishes a small target detection knowledge graph based on the Chinese general knowledge graph and the ImageNet dataset. By modeling the explicit contextual relationships of common targets, it provides more prior knowledge to assist in inferring the target category and location.

[0035] 2. In order to improve the detection effect of small targets while maintaining the recognition speed, this invention adds an Anchor matching module to the YOLOv3 network to make full use of the target-related information in the image, thereby better capturing multi-scale representations, strengthening the receptive field of small targets, and improving the detection effect of blurred and occluded small targets.

[0036] 3. Because this invention uses a small target context knowledge graph and a decoupled head prediction module, it can perform multi-task detection in various application environments, and has better robustness and universality. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the construction of a target context knowledge graph for this invention.

[0038] Figure 2 This is a structural diagram of the improved YOLOv3 model of this invention.

[0039] Figure 3 This is a structural diagram of the feature enhancement unit used in the improved YOLOv3 model in this invention.

[0040] Figure 4 This is a simplified structural diagram of the knowledge graph-based small target detection device of the present invention. Detailed Implementation

[0041] The target context knowledge graph in this invention can add explicit contextual information to the improved YOLOv3 network to help infer the category and location of the target, thereby making full use of the target contextual features in different depth levels of the network and improving the detection accuracy of small targets.

[0042] The invention will be further described below with reference to specific embodiments.

[0043] Example 1:

[0044] like Figure 1As shown, the process of constructing the target context knowledge graph is as follows: Named entity recognition is performed using the BiLSTM+CRF algorithm on the category information of the ImageNet dataset in text format. A Skip-gram model is used to map words to word vectors. Then, the BiLSTM layer obtains the score probability of each word for each label. Finally, the score probability is input into the CRF layer to learn the order dependency information between labels, obtaining the named entity recognition result. Entity links are established between the structured data of the CN-Probase Chinese general concept knowledge graph and the extracted entities. Based on the given entity referents, a set of candidate entity objects is selected from the knowledge base, and then the referents are linked to the correct entity objects through Cosine similarity calculation. An ontology is constructed based on the images from the ImageNet dataset and the obtained factual representations. Schema layer modeling is performed using the RDFS language. For entities with part-of / constraint / dependency relationships, images in the ImageNet dataset where the entities appear simultaneously are selected and labeled. The relative values ​​of the length and width of each target to the length and width of the image are calculated, and these values ​​are filled into the attribute value domain of the corresponding entity. The quality of the formed target context knowledge graph is evaluated, and the knowledge graph is expanded through knowledge reasoning.

[0045] like Figure 2 As shown, the specific structure of the improved YOLOv3 model is as follows: the improved network includes a backbone network module, a feature pyramid module, an anchor matching module, and a decoupled head prediction module. The backbone network module uses a DarkNet53 network with fully connected layers removed for feature extraction, and uses the last three feature maps for prediction. The feature pyramid module upsamples small feature maps with a stride of 2, and then concatenates them with larger feature maps to fuse features. The anchor matching module first generates feature maps of the same size through a feature enhancement unit, and then, based on the aspect ratio data of small and large targets in the knowledge graph, expands the original anchor size using the original small target position as the center point to generate context anchors, and assigns the context anchors to feature layers with larger receptive fields. The decoupled head prediction module first uses 1*1 convolutions for dimensionality reduction, then obtains the classification prediction result of the layer through a classification branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function. It then obtains the confidence prediction result of the layer through a confidence branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function, and finally obtains the coordinate prediction result of the layer through a coordinate branch containing two 3*3 convolutions and one 1*1 convolution. Finally, the results of all layers are concatenated and transposed to obtain the final output.

[0046] like Figure 3As shown, the specific structure of the feature enhancement unit used is as follows: 1*1 convolution and upsampling are applied to the upper feature map, and it is concatenated with the current feature map. Then, a dilated convolution with a kernel of 3*3 and a dilation rate of 3 is used to capture multi-scale representations and enhance the sensory field.

[0047] This embodiment takes face detection as an example, combined with Figure 1 , Figure 2 , Figure 3 The specific workflow of the knowledge graph-based small target detection method of the present invention is described below:

[0048] S1, Construct a target context knowledge graph based on CN-Probase Chinese general concept knowledge graph and ImageNet image dataset;

[0049] S2, in the target context knowledge graph, find related entities based on the "face" category to obtain explicit contextual information to assist in detection;

[0050] S3. Build an improved YOLOv3 small object detection model using the explicit context information obtained in step S2.

[0051] S4: Input the target image into the constructed small target detection model, perform detection processing on the image, and output the detection results.

[0052] In step S1, the specific process of constructing the target context knowledge graph based on the CN-Probase Chinese general concept knowledge graph and the ImageNet image dataset is as follows:

[0053] S11. Named entity recognition is performed on the category information of the ImageNet dataset in text format using the BiLSTM+CRF algorithm: First, the Skip-gram model is used to map words to word vectors. Then, the BiLSTM layer is used to obtain the score probability of each word for each label. Finally, the score probability is input into the CRF layer to learn the order dependency information between labels and obtain the named entity recognition result.

[0054] S12, link the structured data of CN-Probase Chinese General Concept Knowledge Graph with the entities extracted in step S11: select a set of candidate entity objects from the knowledge base based on the given entity references, and then link the references to the correct entity objects through Cosine similarity calculation.

[0055] S13. Construct an ontology based on the images in the ImageNet dataset and the factual representations obtained in step S12: Model the schema layer using the RDFS language. For entities with part-of / constraint / dependency relationships, select images in the ImageNet dataset where the entities appear simultaneously, and perform target annotation. Calculate the relative values ​​of the length and width of each target to the length and width of the image, and fill the attribute value range of the corresponding entity.

[0056] S14: Perform a quality assessment on the formed target context knowledge graph, and expand the knowledge graph through knowledge reasoning.

[0057] In step S2, within the target context knowledge graph, related entities are searched based on the "face" category to obtain explicit contextual information to assist in detection. Specifically, SPARQL is used to query entities that have Part-of / constraint / dependency relationships with the "face" entity, for example: "SELECT ?n WHERE { ?s:className'face'. ?s:Partof ?o. ?o:className ?n}". The ratio of width and height attributes between entities is calculated. For example, based on the width w1 and height h1 of the "face" entity and the width w2 and height h2 of the queried "person" entity, rate_w = w1 / w2 and rate_h = h1 / h2.

[0058] In step S3, the specific steps for building a face detection model using explicit contextual information are as follows:

[0059] S31. The original face dataset is divided into training set, validation set and test set in an 8:1:1 ratio.

[0060] S32, set the initial training parameters, including: using the DarkNet53 model pre-trained on the ImageNet dataset as the backbone; resizing the image size to 640*640 pixels; epoch to 300; batch size to 16; using the Adam optimizer with an initial learning rate of 0.0001, beta1 to 0.937, beta2 to 0.999, and weight decay of 0.0005; warmup epoch to 3.0, initial warmup momentum to 0.8, and initial warmup bias to 0.1.

[0061] In step S33, the training set is input into the initial weight model in batches for forward propagation to obtain the predicted classification and coordinate regression results. Specifically, the Mosaic method is used for data augmentation, which involves randomly using four images, randomly scaling and cropping them, and then randomly distributing and stitching them together. The Mosaic data augmentation method is turned off in the last 15 batches of training. Simultaneously, based on the assumption that "regions of the same scale and aspect ratio for different small targets have similar contextual features," the location and label information of the "person" target are generated proportionally according to the rate_w and rate_h obtained in step S2, focusing on the original target center.

[0062] In step S34, the loss is calculated and backpropagation is performed to update the model weights. Training ends after 300 epochs, and the model is saved every 30 epochs. Specifically, the classification loss is calculated using multi-class cross-entropy, the confidence loss is calculated using binary cross-entropy, and the coordinate loss is calculated using CIoU Loss for the original "face" category and the generated "person" category labels and prediction results, respectively. The weighted sum is then used as the overall loss.

[0063]

[0064] In cross-entropy loss, M represents the number of classes; if the true class of sample i is c, then y ic Select 1 otherwise select 0; p ic Let be the probability that sample i belongs to category c.

[0065]

[0066] In CIoU loss, b and b gt Let ρ represent the center point of the two rectangles, ρ represent the Euclidean distance between the two rectangles, c represent the distance between the diagonals of the enclosing regions of the two rectangles, a is the weighting coefficient, and v is used to measure the consistency of the relative proportions of the two rectangles.

[0067] S35 yields a converged face detection model.

[0068] In step S3, the improved YOLOv3 network specifically includes a backbone network module, a feature pyramid module, an anchor matching module, and a decoupled head prediction module.

[0069] The backbone network module uses the DarkNet53 network with the fully connected layers removed for feature extraction, and uses the feature maps of the last three layers for prediction.

[0070] The feature pyramid module upsamples the small feature map with a step size of 2, and then concatenates it with the large feature map to fuse features.

[0071] The anchor matching module first generates feature maps of the same size through a feature enhancement unit. Then, based on the rate_w and rate_h obtained in step S2, it expands the original anchor size to generate context anchors and assigns them to feature layers with larger receptive fields. For an original image in the target region... targer For the "face" target, the label of the k-th context anchor is defined as:

[0072]

[0073] Where k = 0, 1 represents the regions of "face" and "person" respectively, thre is the IoU threshold, and s pa The step size of the pyramid anchor, anchor i,j Let be the j-th anchor on the i-th feature map, with size s. i .

[0074] The feature enhancement unit performs 1*1 convolution and upsampling on the upper-layer feature map and concatenates it with the current feature map. Then, it uses dilated convolution with a kernel of 3*3 and a dilation rate of 3 to capture multi-scale representations and enhance the sensory field.

[0075] The decoupled head prediction module first uses 1*1 convolutions for dimensionality reduction, then obtains the classification prediction result of the layer through a classification branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function. It then obtains the confidence prediction result of the layer through a confidence branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function, and finally obtains the coordinate prediction result of the layer through a coordinate branch containing two 3*3 convolutions and one 1*1 convolution. Finally, the results of all layers are concatenated and transposed to obtain the final output.

[0076] In step S4, the target image is detected using a small target detection model, specifically: acquiring the target image; inputting the target image into the constructed small target detection model to perform detection processing on the image; and outputting the detection result.

[0077] This invention provides a small target detection method based on a knowledge graph. By constructing a target context knowledge graph, more prior information is added to the traditional YOLOv3 network, enabling the generated context anchor to obtain high-level context features of the small target, thus effectively improving the detection accuracy of small targets in real-world scenarios.

[0078] Example 2:

[0079] like Figure 4As shown, this invention also provides a small target detection device based on a knowledge graph. The device includes at least one processor and at least one memory, coupled together. The memory stores a computer program for a small target detection model constructed as described in Embodiment 1. When the processor executes the computer program stored in the memory, the device performs small target detection. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses in the accompanying drawings are not limited to a single bus or a single type of bus. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk drive, or a USB flash drive, external hard drive, read-only memory, disk, or optical disk, etc.

[0080] The device can be provided as a terminal, server, or other form of device.

[0081] Figure 4 This is a block diagram illustrating an illustrative device. The device may include one or more of the following components: a processing component, a memory, a power supply component, a multimedia component, an audio component, an input / output (I / O) interface, a sensor component, and a communication component. The processing component typically controls the overall operation of the electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. The processing component may include one or more processors to execute instructions to perform all or part of the steps of the methods described above. Furthermore, the processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

[0082] Memory is configured to store various types of data to support the operation of electronic devices. Examples of this data include instructions for any application or method used to operate on an electronic device, contact data, phonebook data, messages, pictures, videos, etc. Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0083] A power supply component provides power to various components of an electronic device. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device. A multimedia component includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0084] The audio component is configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. The I / O interface provides an interface between the processing component and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a power button, and a lock button.

[0085] The sensor assembly includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, the sensor assembly can detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. The sensor assembly may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0086] The communication component is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic device can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0087] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0088] Example 3:

[0089] The present invention also provides a computer-readable storage medium storing a program or instructions for a small target detection model constructed as described in Embodiment 1, wherein when the program or instructions are executed by a processor, the computer performs small target detection.

[0090] Specifically, a system, apparatus, or device may be provided equipped with a readable storage medium on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system, apparatus, or device reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the embodiments described above, therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0091] The aforementioned storage media can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disks or optical disks (such as CD-ROM, CD-R, CD-RW, DVD-20ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tape, etc. The storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0092] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0093] It should be understood that the storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and storage medium can exist as discrete components in a terminal or server.

[0094] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0095] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0097] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A small target detection method based on knowledge graph, characterized in that, Includes the following steps: S1, Construct a target context knowledge graph based on CN-Probase Chinese general concept knowledge graph and ImageNet image dataset; The specific process is as follows: S11. Named entity recognition is performed on the category information of the ImageNet dataset in text format using the BiLSTM+CRF algorithm. First, the Skip-gram model is used to map words to word vectors. Then, the BiLSTM layer is used to obtain the score probability of each word for each label. Finally, the score probability is input into the CRF layer to learn the order dependency information between labels and obtain the named entity recognition result. S12, link the structured data of CN-Probase Chinese General Concept Knowledge Graph with the entities extracted in step S11: select a set of candidate entity objects from the knowledge base according to the given entity references, and then link the references to the correct entity objects through Cosine similarity calculation; S13. Construct an ontology based on the images in the ImageNet dataset and the factual representations obtained in step S12. Use the RDFS language to perform schema layer modeling. For entities with part-of / constraint / dependency relationships, select images in the ImageNet dataset where the entities appear simultaneously, and perform target annotation. Calculate the relative values ​​of the length and width of each target to the length and width of the image, and fill the corresponding entity's attribute value range with these values. S14, perform a quality assessment on the formed target context knowledge graph, and expand the knowledge graph through knowledge reasoning; S2, in the target context knowledge graph, find related entities based on the category information and text description of the small targets in the training set to obtain explicit context information to assist in detection; S3, based on the YOLO v3 network model, an improvement is made, and a small target detection model is built using the explicit context information obtained in step S2; S4. Input the small target image into the small target detection model built in step S3, perform detection processing on the image, and output the detection results.

2. The small target detection method based on knowledge graph as described in claim 1, characterized in that, Step S2 specifically involves: for small target categories that are easily missed in the training set, using the SPARQL language in the target context knowledge graph to query entities that have part-of / constraint / dependency relationships with them, and calculating the ratio of width and height attributes between entities, thus obtaining explicit contextual information to assist in detection.

3. The method for building a small target detection model based on knowledge graphs as described in claim 1, characterized in that, Step S3, which utilizes explicit contextual information to build a small target detection model, involves the following steps: S31, the original dataset is divided into training set, validation set and test set in an 8:1:1 ratio; S32, set the initial training parameters, including: using the DarkNet53 model pre-trained on the ImageNet dataset as the backbone; resizing the image size to 640*640 pixels; epochs to 300; batch size to 16; using the Adam optimizer with an initial learning rate of 0.0001, beta1 of 0.937, beta2 of 0.999, and weight decay of 0.0005; warmup epochs to 3.0, initial warmup momentum of 0.8, and initial warmup bias of 0.

1. S33: The training set is input into the initial weight model in batches for forward propagation to obtain the predicted classification and coordinate regression results. Specifically, the Mosaic method is used for data augmentation, which involves randomly using several images, randomly scaling and cropping them, and then randomly distributing and stitching them together. The Mosaic data augmentation method is turned off in the last 15 batches of training. At the same time, based on the assumption that "regions of the same scale and aspect ratio of different small targets have similar contextual features", the location label and category label of the large target are generated proportionally according to the obtained aspect ratio data of the small and large targets. S34, calculate the loss and perform backpropagation to update the model weights. The training ends when iterates to 300 epochs and the model is saved every 30 epochs. Specifically, the classification loss is calculated by multi-class cross-entropy, the confidence loss is calculated by binary cross-entropy, and the coordinate loss is calculated by CIoU Loss for the labels and prediction results of the original small target and the generated large target, and then the weighted sum is used as the overall loss. S35, test and obtain a converged small target detection model.

4. The method for building a small target detection model based on knowledge graphs as described in claim 1, characterized in that: The improved YOLOv3 network structure includes a backbone network module, a feature pyramid module, an anchor matching module, and a decoupled head prediction module. The backbone network module uses the DarkNet53 network with the fully connected layers removed for feature extraction, and uses the feature maps of the last three layers for prediction. The feature pyramid module upsamples the small feature map with a step size of 2, and then concatenates it with the large feature map to fuse features; The Anchor matching module first generates a feature map of the same size through the feature enhancement unit. Then, based on the obtained aspect ratio data of the small and large targets, it expands the original anchor size with the original small target position as the center point to generate a context anchor and assigns the context anchor to a feature layer with a larger receptive field. The feature enhancement unit performs 1*1 convolution and upsampling on the upper feature map and concatenates it with the current feature map. Then, it uses a dilated convolution with a 3*3 kernel and a dilation rate of 3 to capture multi-scale representations and enhance the receptive field. The decoupled head prediction module first uses 1*1 convolutions for dimensionality reduction, then obtains the classification prediction result of the current layer through a classification branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function. It then obtains the confidence prediction result of the current layer through a confidence branch containing two 3*3 convolutions, one 1*1 convolution, and a sigmoid activation function, and obtains the coordinate prediction result of the current layer through a coordinate branch containing two 3*3 convolutions and one 1*1 convolution. Finally, it concatenates and transposes the results of all layers to obtain the final output.

5. A small target detection device based on knowledge graph, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory being coupled together; the memory stores a computer program for a small target detection model constructed as described in any one of claims 1 to 4; when the processor executes the computer program stored in the memory, the device enables small target detection.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions for constructing a small target detection model as described in any one of claims 1 to 4, wherein when the program or instructions are executed by a processor, the computer performs small target detection.

Images