Target recognition method

Abstract

This invention relates to a target recognition method, comprising: S100, constructing a target knowledge graph based on intelligence information; S200, augmenting the target knowledge graph based on sample information to obtain an augmented target knowledge graph; S300, training a neural network based on the augmented target knowledge graph; S400, inputting a remote sensing image into the trained neural network and outputting candidate categories and location information; S500, converting the candidate categories and location information into fields, and generating entity and relation representations based on the remote sensing image; S600, constructing query logic based on the result of step S500; and S700, converting the query logic into a query language, and filtering target types based on the augmented target knowledge graph. This invention can effectively reduce the dependence of deep learning target recognition algorithms on training, improve the accuracy of deep learning algorithms, and reduce the time cost of algorithm debugging and recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more specifically to a target recognition method. Background Technology

[0002] Remote sensing images are characterized by their wide coverage, large information content, and short observation cycle. The rapid development of remote sensing technology and its important characteristics have led to increasing attention and research on it over the past decade. Target detection and identification are crucial tasks in remote sensing image analysis and a current research focus, holding significant importance for resource surveys, natural disaster monitoring, and military target location.

[0003] Traditional object detection and recognition methods involve training an object detector, primarily by finding and using a superior feature set that effectively distinguishes the target from the background. This feature set is then fed into a classifier for classification, such as Gabor or Histogram of Oriented Gradients (HOG). Therefore, the design of the feature extraction process is a direct factor influencing classification performance.

[0004] In recent years, deep learning methods, such as convolutional neural networks, have become very popular in the field of computer vision due to their effectiveness in feature learning. The emergence of deep learning has revolutionized the traditional process of manually extracting features, allowing images to be described by learning features layer by layer through unsupervised or supervised methods.

[0005] However, most image detection and classification methods rely solely on the image itself, neglecting its semantic information and external intelligence. Current remote sensing image target recognition methods primarily focus on extracting target features from the image itself. Deep learning-based target recognition processes first construct and train a sample set to extract features, then perform target recognition. However, deep learning target recognition algorithms are highly dependent on the quantity and quality of the sample set and the neural network structure; otherwise, the accuracy and correctness of target recognition will decrease. Furthermore, deep learning algorithms are mostly end-to-end, requiring significant time for parameter tuning. Summary of the Invention

[0006] In view of this, the present invention aims to propose a target recognition algorithm to solve the problems of high dependence on training, poor accuracy of learning algorithms, and high time cost of algorithm debugging and recognition in current deep learning algorithms.

[0007] This invention provides a target recognition method, the target recognition method comprising:

[0008] S100, constructing a target knowledge graph based on intelligence information;

[0009] S200, augment the target knowledge graph based on the sample information to obtain an augmented target knowledge graph;

[0010] S300, Train a neural network based on the augmented target knowledge graph;

[0011] S400 inputs remote sensing images into the trained neural network and outputs candidate categories and location information;

[0012] S500, the candidate categories and the location information are converted into fields, and a representation of entities and relationships is generated based on the remote sensing image;

[0013] S600, construct the query logic based on the result of step S500;

[0014] S700, the query logic is transformed into a query language, and the target type is filtered according to the augmented target knowledge graph.

[0015] In a preferred embodiment of the present invention, the sample information in step S200 is image information.

[0016] In a preferred embodiment of the invention, the augmentation includes at least one of flipping, translating, mirroring, adding noise, or rotating.

[0017] In a preferred embodiment of the present invention, the neural network is a Faster-RCNN network, and step S300 further includes:

[0018] Modify the category prediction module of the Faster-RCNN network structure so that the output results are changed to the three candidate categories with the highest probability ranking.

[0019] In a preferred embodiment of the present invention, the output results in step S400 include the top three candidate categories of targets in the remote sensing image and the bounding box coordinates of each target.

[0020] In a preferred embodiment of the present invention, step S500 further includes:

[0021] The fields are combined with the spatial location and time of the remote sensing image to perform entity recognition and disambiguation of named text, and entity relationships are mapped to generate representations of entities and relationships.

[0022] In a preferred embodiment of the present invention, each candidate category generated in step S500 is treated as an entity.

[0023] In a preferred embodiment of the present invention, mapping entity relationships to generate representations of entities and relationships includes:

[0024] The relationships generated by the identification and disambiguation are mapped to the entity attributes or entity relationships of the knowledge base of the augmented target knowledge graph, thus constructing a triple representation of entities and relationships.

[0025] In a preferred embodiment of the present invention, step S600 includes:

[0026] The entity is queried based on attributes, and the candidate categories of the target to be identified are filtered. Incorrect triple representations are deleted, and the target categories are further filtered by combining the information of the remote sensing image and the reasoning results of the augmented knowledge graph. The relationship between the remaining candidate categories of the target is then determined.

[0027] In a preferred embodiment of the present invention, the query logical expression in step S600 is:

[0028] hx < length <h+x;

[0029] wx < width <w+x;

[0030] Where h is the length of the predicted bounding box, w is the width of the predicted bounding box, and x is the error between the predicted bounding box and the ground truth target.

[0031] The target recognition method of this invention can improve the accuracy of target recognition by introducing the analysis results of knowledge graphs. It changes the classifier in the deep learning network that predicts the category to output the result based on confidence, and transforms the recognition result of deep learning into semantic information. Combined with the representation and reasoning capabilities of knowledge graphs, the method can effectively reduce the impact of the sample set on the accuracy and correctness of the deep learning network, and can integrate sample information from other sources to improve the efficiency and accuracy of image recognition. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the target recognition method according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic block diagram illustrating the principle of the target recognition method according to an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of image augmentation according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the improved Faster-RCNN network according to an embodiment of the present invention. Detailed Implementation

[0037] The description of the embodiments in this specification should be taken in conjunction with the accompanying drawings, which should form part of the complete specification. In the drawings, the shape or thickness of the embodiments may be exaggerated and may be indicated in a simplified or convenient manner. Furthermore, parts of the various structures in the drawings will be described separately; it is worth noting that elements not shown in the figures or not described in words are in a form known to those skilled in the art.

[0038] The descriptions of the embodiments herein, including any references to directions and orientations, are for ease of description only and should not be construed as limiting the scope of the invention. The following description of preferred embodiments involves combinations of features, which may exist independently or in combination; the invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.

[0039] like Figure 1-2 As shown, this embodiment of the invention relates to a target recognition method. The following detailed description of this embodiment, using ship target recognition based on space-based optical remote sensing images as an example, specifically includes:

[0040] S100, constructs a target knowledge graph based on intelligence information.

[0041] In this embodiment, for ship targets, intelligence information includes, but is not limited to, the ship target's designation and location. By integrating external intelligence information to construct a knowledge graph, the sample capacity can be effectively increased, thereby improving the accuracy of results when using deep learning network algorithms subsequently.

[0042] S200, augment the target knowledge graph based on the sample information to obtain an augmented target knowledge graph.

[0043] like Figure 3 As shown, in this embodiment, the sample information is image information, and the augmentation methods for the sample information include at least one of flipping, translating, mirroring, adding noise, or rotating. Preferably, the sample information can be augmented by combining the aforementioned five methods to maximize the expansion of the sample information in the knowledge graph, which is beneficial to improving the accuracy of results when using deep learning network algorithms.

[0044] S300, Train the neural network based on the augmented target knowledge graph.

[0045] like Figure 4 As shown, in this embodiment, the neural network is a Faster-RCNN network. By changing the category prediction module (Softmax regression) of the Faster-RCNN network structure, the output result is not a specific category, but the output result is changed to the three candidate categories with the highest probability ranking.

[0046] The S400 takes remote sensing images as input to the trained neural network and outputs candidate categories and location information.

[0047] In this embodiment, for ship targets, the network model outputs the top three candidate categories of the targets in the image, as well as the bounding box coordinates of each target.

[0048] S500, the candidate category and the location information are converted into fields, and a representation of entities and relationships is generated based on the remote sensing image.

[0049] In this embodiment, each candidate category generated in step S400 is treated as an entity. The result of step S400 is combined with image metadata to transform it into entities and relationships:

[0050] a. Perform named entity recognition and disambiguation: Based on the candidate category of the target, perform entity disambiguation processing, such as "Arleigh Burke-class guided-missile frigate" should be mapped to "Arleigh Burke-class guided-missile frigate".

[0051] b. Perform entity relation mapping, mapping the relations identified and disambiguated in step a to entity attributes or entity relations in the knowledge base of the augmented target knowledge graph. Since the entities and relations formed by the detection and image features in step a have already been processed and can be directly mapped, the generated relations need to be mapped to entity attributes or entity relations in the augmented target knowledge graph. Based on the coordinate information of the remote sensing image, determine the sea area or specific port; if the coordinates are at sea, the relation is "navigating"; if the coordinates are at a port, the relation is "docking".

[0052] Estimate the length and width of the target based on the size of the output prediction box.

[0053] Construct entity-relation triples (entity-relation-entity).

[0054] For example, the output of a deep learning object recognition algorithm is:

[0055] (Nimitz-class aircraft carrier, Ford-class aircraft carrier, Asagiri-class multi-purpose destroyer) [x1,y1,x2,y2];

[0056] (Arleigh Burke-class guided-missile destroyer, Kidd-class guided-missile destroyer, Perry-class guided-missile frigate) [x1,y1,x2,y2];

[0057] The image coordinates of the remote sensing image, after being queried, are "Yokousuka", and the time is t1 = month x day xx hour xx minute;

[0058] Based on the size of the predicted bounding box in the output, the length and width attributes of the target are estimated and denoted as [w1, h1] and [w2, h2], respectively. Then, the triplet of each target in the image is represented in the knowledge graph as follows:

[0059] Target 1: (Target 1, length, h1);

[0060] (Target 1, Width, w1);

[0061] (Objective 1, Time, t1);

[0062] (Nimitz-class aircraft carrier, docked at Yokosuka);

[0063] (Ford-class aircraft carrier, docked at Yokosuka);

[0064] (Asagiri-class multipurpose destroyer, docked at Yokosuka);

[0065] Objective 2: (Objective 2, length, h2);

[0066] (Target 2, Width, w2);

[0067] (Objective 2, Time, t1);

[0068] (Arleigh Burke-class guided-missile destroyer, docked at Yokosuka);

[0069] (Kidd-class guided-missile destroyer, docked at Yokosuka);

[0070] (Perry-class guided-missile frigate, docked at Yokosuka).

[0071] S600: Construct query logic based on the results of step S500.

[0072] In this embodiment, it specifically includes:

[0073] (1) Determine the possible types of the target based on its attributes. Since the predicted bounding box has an error compared to the actual target and is not the actual target's length and width, the error x needs to be added when performing length and width attribute retrieval:

[0074] Query type Logical expressions Query entities by attribute h - x < length < h + x AND w - x < width < w + x

[0075] (2) Based on the query results, filter the candidate categories of the target to be identified and delete the incorrect triplet representations.

[0076] (3) Based on the remaining categories after filtering and the reasoning results of the augmented target knowledge graph, determine the location of the candidate categories according to the detection time. Because the target knowledge graph is constructed based on intelligence information, entities may be accurate down to the ship name, and there may be multiple locations of the same category at the same time in the knowledge graph.

[0077] (4) Based on the location information of the remote sensing image, filter the ship target categories from the results of (3) above.

[0078] (5) Further, based on the remaining candidate categories after screening, determine the possible relationships between different target candidate categories, thereby improving the accuracy of target identification and verifying the correctness of the search.

[0079] S700, the query logic is transformed into a query language, and the target type is filtered according to the augmented target knowledge graph.

[0080] In this embodiment, the query logic is converted into a query language, input into the knowledge graph system for querying, and finally the corresponding target to be identified is obtained.

[0081] It should be understood that this embodiment uses the identification of ship targets based on space-based optical remote sensing images as an example, and is not intended to limit this embodiment. Those skilled in the art can apply the method of this invention to the identification of other remote sensing systems and other target objects.

[0082] In summary, this invention combines deep learning target recognition algorithms with knowledge graphs to propose an improved target recognition method. First, intelligence is integrated to construct a knowledge graph. Then, deep learning target recognition algorithms are used to extract features of targets in remote sensing images, combining these features with the semantic information of the images themselves to transform them into semantic information. Finally, the acquired information is fed into the knowledge graph for target recognition. Target recognition of remote sensing images, based on deep learning target recognition algorithms and associated with external intelligence knowledge graphs, can effectively improve the accuracy of target recognition. Furthermore, since remote sensing images have fewer datasets than everyday images, and deep learning methods heavily rely on the quality and quantity of training data, combining deep learning with knowledge graphs can effectively reduce the cost of creating training sets for deep learning and improve training efficiency. Therefore, this invention can effectively reduce the dependence of deep learning target recognition algorithms on training, improve the accuracy of deep learning algorithms, and reduce the time cost of algorithm debugging and recognition through intelligence integration.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target recognition method characterized by, The target recognition method comprises: S100, constructing a target knowledge graph according to intelligence information, the intelligence information comprising code information and position information of a ship target; S200, augmenting the target knowledge graph according to sample information to obtain an augmented target knowledge graph, the sample information being image information; S300, training a neural network according to the augmented target knowledge graph; S400, inputting a remote sensing image into the trained neural network and outputting candidate categories and position information; S500, converting the candidate categories and the position information into fields to generate representations of entities and relationships according to the remote sensing image, each candidate category generated in this step being an entity; According to coordinate information of the remote sensing image, it is determined that the entity is in a sea area or a specific port; if the coordinates are on the sea, the relationship is "sailing", and if the coordinates are a port, the relationship is "docking"; S600, constructing a query logic according to the result of step S500, comprising: According to the attributes, the entities are queried, and the candidate categories of the target to be recognized are screened, and the incorrect triple representations are deleted, the target categories are screened in combination with the information of the remote sensing image and the reasoning result of the augmented target knowledge graph, and the relationships between the remaining candidate categories of the target are determined; wherein the triple representation is: entity-relation-entity; The attributes refer to the length, width and time of the target; According to the size of the output prediction frame, the length and width of the target are estimated; S700, converting the query logic into a query language and screening the target types according to the augmented target knowledge graph.

2. The object recognition method of claim 1, wherein, The augmentation comprises at least one of flipping, translation, mirroring, adding noise or rotation.

3. The object recognition method of claim 1, wherein, The neural network is a Faster-RCNN network, and step S300 further comprises: Changing a category prediction module of the Faster-RCNN network structure so that the output result is changed to the top three candidate categories in terms of possibility.

4. The object recognition method of claim 3, wherein, The output result in step S400 comprises the top three candidate categories of the target in the remote sensing image and the boundary box coordinates of each target.

5. The object recognition method of claim 4, wherein, Step S500 further comprises: Performing entity recognition and disambiguation on the fields in combination with the spatial position and time of the remote sensing image, and mapping the entity relationships to generate representations of entities and relationships.

6. The object recognition method of claim 5, wherein, Mapping the entity relationships to generate representations of entities and relationships comprises: Mapping the relationships generated by the recognition and disambiguation to the entity attributes or entity relationships of the knowledge base of the augmented target knowledge graph to construct triple representations of entities and relationships.

7. The object recognition method of claim 6, wherein, The query logic expression in step S600 is: h-x < length < h+x; w-x < width < w+x; Wherein, h is the length of the prediction frame, w is the width of the prediction frame, and x is the error between the prediction frame and the real target.

Images