Atlas creation and comparison methods
Patent Information
- Authority / Receiving Office
- TW · TW
- Patent Type
- Patents
- Current Assignee / Owner
- FENG CHIA UNIVERSITY
- Filing Date
- 2024-11-11
- Publication Date
- 2026-08-01
AI Technical Summary
Existing object recognition models fail to establish correlations between unknown objects and known objects, leading to incomplete image classification when unknown objects are present.
A method involving constructing a test map with complex attribute nodes, establishing relationships with known graphs, calculating main category node values, and using a normalized exponent function to determine total correlation weights, enabling the neural network to understand and process unknown objects.
Enables the neural network to establish associations between unknown and known objects, allowing for accurate image classification and subsequent processing steps.
Smart Images
Figure TWG2TB001903678_001 
Figure TWG2TB001903678_002 
Figure TWG2TB001903678_003
Abstract
Description
Technical Field
[0001] This invention relates to neural network technology, and in particular to a method for establishing an unknown knowledge graph and comparing its correlation with other knowledge graphs. Prior Technology
[0002] Existing neural network detection methods for a test image typically involve inputting the test image into a trained object recognition model. This model compares the test image with a large amount of stored data to determine the category of the test image. The detection method includes two levels: one is to mark or separate objects in the test image that belong to the same attribute, and the other is to compare the marked or separated objects with known objects in the object recognition model to establish associations, thereby defining the category of the objects.
[0003] However, existing object recognition models have a problem with object identification: when the image to be tested contains an unknown object that the model cannot identify, the model cannot establish a correlation between the unknown object and the known objects that the model can identify. The unknown object must then be manually labeled to classify the image. For example, if the image contains three objects, and the model only identifies two, the remaining object is considered undetected. Naturally, the model will not detect the remaining object; it will treat it as the unknown object (the undetectable object). Since existing object recognition models cannot understand that the image to be tested contains an unknown object, they will not take any action on the unknown object, thus failing to accurately identify the image to be tested. Summary of the Invention
[0004] When identifying a test image containing unknown objects, existing object recognition models cannot establish a correlation between the unknown objects and identifiable known objects, causing them to take no action on the unknown objects. Therefore, this invention proposes a map establishment and comparison method, comprising: A test image is identified using an object recognition model to generate multiple bounding boxes; Based on the complex detection box, a test map is constructed corresponding to the image to be tested. The test map contains complex attribute nodes, and each complex attribute node corresponds to a complex detection box. Establish the relationship between the target graph and a group of known graphs, where the group of known graphs contains a complex number of known graphs, and each known graph contains a principal category node value; Calculate the main category node value of each known map based on the relationship between the map to be tested and the known map group; Input the principal category node values of each known graph into a normalized exponent (Softmax) function to calculate the complex total correlation weight between the graph to be tested and each of the known graphs.
[0005] When the object recognition model identifies the image to be tested that may contain unknown objects, the map establishment and comparison method of the present invention can establish an association between the map to be tested corresponding to the image to be tested and each of the known maps by using the complex total association weight value, thereby enabling the neural network to understand the potential relationship between the image to be tested and each of the known maps and to take subsequent processing steps. Simple Explanation of the Diagram
[0006] Figure 1: Circuit block diagram of the detection device for implementing the spectrum establishment and comparison method of the present invention. Figure 2: Flowchart of the method for establishing and comparing the atlases of the present invention. Figure 3: Schematic diagram of the spectrum to be tested in the spectrum establishment and comparison method of the present invention. Figure 4: Schematic diagram of the spectrum to be tested and the known spectrum group in the spectrum establishment and comparison method of the present invention. Figure 5: Flowchart of the method for establishing and comparing the atlases of the present invention. Figure 6: A schematic diagram of the target graph and the known graph group in the graph establishment and comparison method of the present invention, wherein at least two of the complex correlation weight edges are connected to one of the category nodes 31. Figure 7: Schematic diagram of the category node algorithm of the map establishment and comparison method of the present invention. Figure 8: A schematic diagram of the numerical calculation of category nodes in the map establishment and comparison method of the present invention, wherein the category node is connected to both the category weight edge and the associated weight edge. Implementation
[0007] To gain a detailed understanding of the technical features and practical effects of the present invention, and to enable its implementation according to the invention, the following detailed description is provided with reference to the embodiments shown in the figures:
[0008] Please refer to Figure 1. The atlas establishment and comparison method of the present invention is executed by a detection device 10. The detection device 10 includes a processing unit 11 and a storage unit 12. The processing unit 11 is connected to the storage unit 12 and can read and write data in the storage unit 12. For example, the processing unit 11 can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), or other processing devices. The storage unit 12 stores the program code of an object recognition model 13 and an image database 14. The object recognition model 13 is connected to the image database 14. The processing unit 11 can read the data in the image database 14 and execute the object recognition model 13. It executes the object recognition model 13 according to the data in the image database 14 and writes the execution result of the object recognition model 13 into the image database 14. The storage unit 12 can be, for example, a hard disk, a memory, or a network attached storage device. This invention is not limited to storage devices such as NAS.
[0009] Please refer to Figure 2. The method for establishing and comparing the spectrum includes steps S10 to S60, and each step is explained below.
[0010] Step S10: The object recognition model 13 identifies a test image to generate multiple bounding boxes. Specifically, the processing unit 11 receives the test image, for example, the test image is pre-stored in the storage unit 12. The processing unit 11 reads (receives) the test image in the storage unit 12 to perform subsequent object recognition calculations. Alternatively, the processing unit 11 has an input / output interface to receive the test image and output the calculation results from outside the detection device 10. The input / output interface can be an integrated bus circuit (I2C), a sequential peripheral interface (SPI), etc., and this invention is not limited thereto. Next, the processing unit 11 reads and executes the object recognition model 13. The object recognition model 13 can perform image segmentation on the test image, that is, classify the multiple pixels in the test image to generate multiple bounding boxes. The set of pixels in each bounding box is a test object.
[0011] The object recognition model 13 generates the multiple detection boxes based on features such as texture, color, edge shape, and size in the image to be tested. The object recognition model 13 can be, for example, a neural network model based on candidate region deep learning (Region-Based Convolutional Neural Network (R-CNN)) or Region-Based Fully Convolutional Networks (R-FCN)), or a neural network model based on recursive deep learning (You Only Look Once (YOLO) or Single Shot MultiBox Detector (SSD)). This invention is not limited to these.
[0012] Step S20: Establish a test map corresponding to the test image based on the complex detection boxes. Specifically, the processing unit 11 establishes a test map 20 as shown in Figure 3 based on the complex detection boxes generated by the object recognition model 13. The test map 20 includes complex attribute nodes 21, and each complex attribute node 21 corresponds to a complex detection box. For example, if the content of the test image is a cat, the complex detection boxes may select the cat as a whole, the cat's head, the cat's eyes, the cat's body, and the cat's paws, etc., and the complex attribute nodes correspond to the test objects. Preferably, before establishing the test map 20, the object recognition model can eliminate redundant detection boxes through a non-maximum suppression (NMS) algorithm.
[0013] As shown in Figure 3, the complex attribute nodes 21 are connected in a hierarchical data tree. Each attribute node 21 includes a primary attribute node 210 and at least one primary attribute node 211. The primary attribute node 210 connects to each of the secondary attribute nodes 211 through an attribute weight edge PE. The value of each attribute weight edge PE is the reciprocal of the number of primary attribute nodes connected to the primary attribute node 210. For example, taking the top-level attribute node 21 of the graph to be tested 20 as the primary attribute node 210, the primary attribute node 210 connects to two secondary attribute nodes 211. The value of the attribute weight edge PE between the primary attribute node 210 and the secondary attribute node 211 is 1 / 2 (the number of secondary attribute nodes 211 is 2, and its reciprocal is 1 / 2). The primary attribute node 210 corresponds to the entire cat, where the primary attribute node 211 corresponds to the head of the cat, and the other secondary attribute node 211 corresponds to the body of the cat.
[0014] It should be noted that in the graph to be tested 20, except for the bottommost attribute node 21 which cannot be used as the main attribute node 210, all other attribute nodes 21 can be used as the main attribute node 210. For example, assuming that the attribute node 21 corresponding to the cat's body is used as the main attribute node 210, the at least one attribute node 211 connected to the main attribute node 210 is, for example, the cat's paw. The number of layers in the graph to be tested 20 is determined by the number of complex detection frames, and this invention is not limited thereto.
[0015] Step S30: Establish the relationship between the test map 20 and a known map group. Specifically, please refer to Figure 4. The processing unit 11 reads a known map group 30 from the storage unit 12. The known map group 30 contains a complex number of known maps. For example, the complex known maps are a first known map 30A, a second known map 30B, and a third known map 30C. Each known map contains a complex number of category nodes 31. The complex number of category nodes 31 are also connected in a hierarchical data structure. Each known map has a main category node value, which is the value of the top-level category node 31 of each known map.
[0016] The method of establishing the known image group 30 is roughly the same as the method of establishing the image to be tested 20. In short, the image database 14 stores a plurality of known images. The processing unit 11 reads each known image and executes the object recognition model 13 to identify each known image and generate a plurality of detection frames for each known image. Then, based on the plurality of detection frames of each known image, the known image corresponding to each known image is established, and the known image of each known image is stored in the storage unit 12 for the processing unit 11 to read and use.
[0017] The relationship between the test spectrum 20 and the known spectrum group 30 can be established by the processing unit 11 through the sub-step process shown in Figure 5.
[0018] Sub-step S31: The processing unit 11 generates complex association weight edges RE between the test graph 20 and one of the complex known graphs through a similarity algorithm. Specifically, each association weight edge RE is connected between one of the complex category nodes 31 and one of the complex attribute nodes 21. For example, as shown in Figure 4, three attribute nodes 21 in the test graph 20 are respectively connected to three category nodes 31 in the first known graph 30A. The similarity algorithm can be, for example, a cosine similarity-based algorithm, a KL divergence-based algorithm, etc., and the present invention is not limited thereto.
[0019] Sub-step S32: The processing unit 11 determines whether the value of each weight edge RE is greater than or equal to a weight threshold. The storage unit 12 stores the weight threshold. The processing unit 11 reads the weight threshold to make the determination. For example, the values of the weight edges RE connected between the three attribute nodes 21 and the three category nodes 31 are 0.75, 0.7 and 0.77 respectively. Assuming that the weight threshold is 0.7, then the values of the weight edges RE are all greater than or equal to the weight threshold.
[0020] Sub-step S33: Retain the associated weight edge RE with a value greater than or equal to the weight threshold as the relationship between the test graph 20 and the known graph group 30. That is, the test graph 20 and the first known graph 30A will be connected by a first associated weight edge with a value of 0.75, a second associated weight edge with a value of 0.7, and a third associated weight edge with a value of 0.77. Conversely, when the value of an associated weight edge RE is less than the weight threshold, the processing unit 11 executes sub-step S34 to remove the associated weight edge RE.
[0021] Furthermore, during the process of establishing the relationship between the test graph 20 and the known graph group 30 by the processing unit 11, as shown in FIG6, when at least two of the complex association weight edges RE are connected to one of the category nodes 31, the processing unit 11 retains the one with the largest value among the complex association weight edges RE to establish the relationship between the test graph 20 and the known graph group 30, and removes the other association weight edges RE connected to the category node 31, that is, a category node 31 is only associated with one attribute node 21 through the association weight edge RE.
[0022] Step S40: The processing unit 11 calculates the main category node value of each known map 30 according to the relationship between the map to be tested 20 and the known map group 30. Specifically, step S40 includes the following sub-steps S41 to S44.
[0023] Sub-step S41: The processing unit 11 calculates a temporary primary category node value for each known graph based on the values of the complex association weight edges, the complex common edges, and a category node algorithm. Specifically, as shown in Figure 4, the complex category nodes 31 of the complex known graph are connected through the complex common edges CE, that is, each end of the common edge CE is connected to a category node 31, and the complex category nodes 31 in each known graph are connected through a category weight edge SE. The category node algorithm is that the value of the category node 31 outside the bottom layer of each known graph is the sum of the values of the category nodes 31 connected to the bottom of the category node 31 multiplied by the reciprocal of the number of category nodes 31 connected to the category node 31.
[0024] For example, referring to Figure 7, a first category node 311 is connected to a second category node 312 and a third category node 313 at its bottom. The first category node 311 and the second category node 312 are connected through the category weight edge SE, and the first category node 311 and the third category node 313 are connected through the category weight edge SE. Since the number of category nodes 31 connected to the first category node 311 is 2, the value of each category weight edge SE is 1 / 2. Therefore, the number of first category nodes is the value of the second category node 312 multiplied by the value of the category weight edge SE plus the value of the third category node 313 multiplied by the value of the category weight edge SE, that is, the sum of the second category node 312 and the third category node 313 multiplied by the value of the category weight edge SE (the reciprocal of the number of category nodes 31 connected to the first category node 311).
[0025] Furthermore, referring to Figure 8, when one of the complex category nodes 31 (e.g., the first category node 311) is simultaneously connected to the category weight edge SE and the association weight edge RE, the value of the category node 31 (the first category node) is the larger of a category product and an association product, wherein the category product is the value of the category weight edge SE multiplied by the value of the category node 31 connected to the other end of the category weight edge SE, and the association product RE is the value of the association weight edge RE multiplied by the value of the attribute node 21 connected to the other end of the association weight edge RE.
[0026] Therefore, as shown in Figure 4, the value of the category node 31 in the first known graph 30A can be calculated based on the value of the attribute node 21 and the value of the complex association weight edge in the graph to be tested 20. Then, the value of the temporary main category node in the first known graph 30A is calculated through the category node algorithm. Based on the value of the common edge connecting the first known graph 30A with the second known graph 30B and the third known graph 30C, the value of the category node 31 in the second known graph 30B and the third known graph 30C is calculated. Then, the values of the temporary main category nodes in the second known graph 30B and the third known graph 30C are calculated respectively.
[0027] Sub-step S42: The processing unit 11 determines a data transmission direction based on the size of the temporary principal category node value of each known graph, wherein the data transmission direction is from a source graph to a sink graph. The known graph with a larger temporary principal category node value is defined as the source graph, and the known graph with a smaller temporary principal category node value is defined as the sink graph.
[0028] For example, referring to Figure 4, the first known graph 30A connects the second known graph 30B and the third known graph 30C through the complex common edge CE. Assuming that after calculation in sub-step S41, the temporary principal category node value of the first known graph 30A is greater than the temporary principal category node value of the second known graph 30B and the temporary principal category node value of the third known graph 30C, then the first known graph 30A is the source graph, and the second known graph 30B and the third known graph 30C are the sink graphs.
[0029] Sub-step S43: The processing unit 11 updates the values of each category node 31 in the sink graph according to the data transmission direction, and then calculates the main category node value of the source graph. Specifically, one end of each common edge connects to a source node and the other end connects to a sink node. The category node 31 in the source graph is the source node, and the category node 31 in the sink graph is the sink node. The value of the sink node is the value of the source node multiplied by the value of the common edge connecting the sink node and the source node. Then, the main category node value of the sink graph is calculated according to the values of each category node in the sink graph (the value of the sink node) and the category node algorithm. The temporary main category node value of the source node is used as the main category node value of the source graph.
[0030] For example, referring to Figure 4, the first known graph 30A includes a first source node 31A and a second source node 31B, the second known graph 30B includes a first sink node 31C, and the third known graph 30C includes a second sink node 31D. The first source node 31A is connected to the first sink node 31C through a common edge, and the second source node 31B is connected to the second sink node 31D through the common edge. The value of the first sink node 31C is the value of the first source node 31A multiplied by the value of the common edge, and the value of the second sink node 31D is the value of the second source node 31A. The value of source node 31B is multiplied by the value of the common edge. Then, the values of other nodes of the same category 31 in the second known graph 30B are updated through the first sink node 31C and the category node algorithm. This is used to calculate the main category node value of the second known graph 30B. The values of other nodes of the same category 31 in the third known graph 30C are updated through the second sink node 31D and the category node algorithm. This is used to calculate the main category node value of the third known graph 30C. The temporary category node value of the first known graph 30A is retained as its main category node value.
[0031] Sub-step S44: Determine whether the sink graph contains the common edge. If the sink graph contains the common edge, execute sub-step S42 again to update the main category node value of the sink graph. If the sink graph does not contain the common edge, execute step S50 according to the main category node value of each known graph.
[0032] Please refer to Figure 4 again. The second known graph 30B and the third known graph 30C contain a common edge. Assuming that after the update in sub-step S43, the value of the main category node of the second known graph 30B is less than the value of the main category node of the third known graph 30C, then the third known graph 30C is the new source graph, and the second known graph 30B is the new sink graph. Sub-step S42 is executed again to update or retain the value of the main category node of the second known graph 30B and the value of the main category node of the third known graph 30C respectively.
[0033] Step S50: Input the value of the main category node of each known graph into a normalization exponent (Softmax) function to calculate the complex total correlation weight value between the test graph 20 and each known graph. Specifically, as shown in Figure 4, the top-level attribute node 21 of the test graph 20 is connected to the top-level category node 31 of the first known graph 30A, the top-level category node 31 of the second known graph 30B, and the top-level category node 31 of the third known graph 30C through a first total correlation weight edge TRE1, a second total correlation weight edge TRE2, and a third total correlation weight edge TRE3. The values of the first total correlation weight edge TRE1, the second total correlation weight edge TRE2, and the third total correlation weight edge TRE3 are the complex total correlation weight values. Since the corresponding domain of the normalization exponent function is 0~1, each total correlation weight value is a value between 0 and 1.
[0034] Subsequently, the processing unit 11 may also execute a step S60. In step S60, the processing unit 11 determines the numerical range to which the image to be tested belongs based on the complex total correlation weight value, and stores the image to be tested in a designated directory of the image database 14 corresponding to the numerical range. Specifically, the image database 14 may first divide the data into complex numerical ranges based on the corresponding domain of the normalized exponential function. For example, the value 0 to 1 may be divided into 10 numerical ranges, namely, the value 0 to 0.099 is a first numerical range, the value 0.1 to 0.199 is a second numerical range, ... the value 0.9 to 0.999 is a tenth numerical range. When the total correlation weight value is between the upper limit and the lower limit of one of the numerical ranges, the total correlation weight value is within that numerical range.
[0035] As mentioned above, after the processing unit 11 calculates in steps S10 to S50, assuming that the total association weight value of the first total association weight edge TRE1 is 0.698, the total association weight value of the second total association weight edge TRE2 is 0.144, and the total association weight value of the third total association weight edge TRE3 is 0.158, then the similarity between the test map 20 and the first known map 30A is the highest. The test image corresponding to the test map 20 will be stored in the image database 14 in the specified directory corresponding to a seventh numerical range (value 0.6~0.7), and the specified directory also corresponds to the first known map 30A.
[0036] The atlas establishment and comparison method of the present invention is implemented by a processing unit 11. The processing unit 11 executes an object recognition model 13, which recognizes an image to be tested to generate complex detection boxes. The processing unit 11 establishes a test atlas 20 of the image to be tested based on the complex detection boxes, and establishes the relationship between the test atlas 20 and a known atlas group 30. The processing unit 11 calculates a main category node value of each known atlas in the known atlas group 30 based on the relationship between the test atlas 20 and the known atlas group 30, and assigns the main category of each known atlas to the main category. Each node value is input to a normalized exponential function to calculate the complex total correlation weight value between the test image 20 and each of the known images. When the object recognition model 13 identifies the test image that may contain unknown objects, this method can establish a correlation between the test image 20 corresponding to the test image and each of the known images through the complex total correlation weight value, thereby enabling the neural network to understand the potential relationship between the test image and each of the known images and to take subsequent processing steps. For example, the processing unit 11 can store the test image in a specified directory of an image database 14.
[0037] In summary, this description merely illustrates the implementation methods or embodiments of the technical means employed by the present invention to solve the problem, and is not intended to limit the scope of the present invention patent. That is, all changes and modifications that conform to the meaning of the text of the present invention patent application, or are equivalent to those made in accordance with the scope of the present invention patent, are covered by the scope of the present invention patent.
[0038] 10: Testing equipment 11: Processing Unit 12: Storage Unit 13: Object Recognition Model 14: Image Database 20: The spectrum to be tested 21: Attribute Node 210: Main attribute node 211: Secondary attribute node 30: Known atlas group 30A: First known map 30B: Second known map 30C: Third known spectrum 31: Category Node 31A: First source node 31B: Second source node 31C: First Node 31D: Second sink node 311: First-class node 312: Second category node 313: Third-category node CE: Common Edge PE: Attribute Weighted Edge RE: Associated Weighted Edges SE: Category Weight Edge TRE1: First total association weight edge TRE2: Second Total Association Weight Edge TRE3: Third Total Association Weight Edge
Claims
1. A method for establishing and comparing a graph, executed by a detection device, comprising: identifying a test image through an object recognition model to generate complex detection boxes; establishing a test graph corresponding to the test image based on the complex detection boxes, the test graph containing complex attribute nodes, each of the complex attribute nodes corresponding to the complex detection boxes; establishing a relationship between the test graph and a known graph group, wherein the known graph group contains complex known graphs, each of the known graphs containing a principal category node value; calculating the principal category node value of each of the known graphs based on the relationship between the test graph and the known graph group; inputting the principal category node value of each of the known graphs into a normalization index (Softmax) function to calculate the complex total correlation weight value between the test graph and each of the known graphs; The complex attribute node includes a primary attribute node and at least one primary attribute node. The primary attribute node is connected to each of the primary attribute nodes through an attribute weight edge. The value of each attribute weight edge is the reciprocal of the number of primary attribute nodes connected to the primary attribute node.
2. The image creation and comparison method as described in claim 1 further includes: determining a numerical range to which the image to be tested belongs based on the complex total correlation weight value, and storing the image to be tested in a specified directory of an image database corresponding to the numerical range.
3. The map creation and comparison method as described in Request 1, wherein: Each known graph contains a number of category nodes, and the number of category nodes in each known graph and the number of attribute nodes in the graph to be tested are connected in a hierarchical data tree.
4. The map creation and comparison method as described in Request 1, wherein: A similarity algorithm is used to generate complex association weight edges between the target graph and one of the complex known graphs; it is determined whether the value of each association weight edge is greater than or equal to a weight threshold; the association weight edges with values greater than or equal to the weight threshold are retained as the relationship between the target graph and the group of known graphs, wherein each known graph contains complex category nodes, and each association weight edge connects one of the complex category nodes and one of the complex attribute nodes.
5. The map creation and comparison method as described in request item 4, wherein: When at least two of the complex association weight edges are connected to one of the complex category nodes, the edge with the largest value among the complex association weight edges is retained to establish the relationship between the graph to be tested and the known graph.
6. The map creation and comparison method as described in request item 4, wherein: The complex class nodes of the known complex graph are connected by complex common edges; a temporary principal class node value for each known graph is calculated based on the values of the complex association weight edges, the values of each common edge, and a class node algorithm; a data transmission direction is determined based on the magnitude of the temporary principal class node value of each known graph, which is from a source graph to a sink graph, wherein the known graph with the larger temporary principal class node value is defined as the source graph, and the known graph with the smaller temporary principal class node value is defined as the sink graph.
7. The map creation and comparison method as described in request item 6, wherein: Update the values of each category node in the sink graph according to the data transmission direction; calculate the value of the main category node of the sink graph according to the values of each category node in the sink graph and the category node algorithm, and use the temporary main category node value of the source graph as the main category node value.
8. The map creation and comparison method as described in request item 6, wherein: Each common edge connects a source node at one end and a sink node at the other end. The node of the same category in the source graph is the source node, and the node of the same category in the sink graph is the sink node. The value of the sink node is the value of the source node multiplied by the value of the common edge connecting the sink node and the source node.
9. The map creation and comparison method as described in request item 6, wherein: Each of the complex category nodes in the known graph is connected by a category weight edge. When one of the complex category nodes is connected to both the category weight edge and the association weight edge, the value of the category node is the larger of a category product and an association product. The category product is the value of the category weight edge multiplied by the value of the category node connected to the other end of the category weight edge, and the association product is the value of the association weight edge multiplied by the value of the attribute node connected to the other end of the association weight edge.